I didn’t get why 12?

That’s really interesting. Yeah I’d love for you to send me the code you used, I’ll play around with it myself. If you don’t mind I might even do a separate post about it (if I can fit it in that is, so much to do so little time :)).

I decided to try comparing the performance of my approach to yours. I tried your timing method but was not satisfied because on my slowish hardware the resolution appears to be about 15.625 microseconds and thus on a single pass I could not distinguish the relative speeds of each approach. I then changed to Benchmark only to discover that you had already blogged about it! I think Benchmark gives a better picture of the relative performances. I ran 3 types of lists through the benchmarking process just as you had done: raw integer lists, wrapped lists and wrapped lists with skips. I stored the integer values of the list members that exist in the intersection in the output list. This made it easy to verify that each intersection method returned the same output. I created the lists once and stored them as yaml to allow repeated testing.

The results surprised me:

    list1 size: 1064      list2 size: 4014      intersect list size: 416
    1000 passes:
		Enumerator Based Intersect	        Skorks Rubyish Intersect
		user	system	 total	 real		user	system	 total	 real
raws:	21.59	0.02	21.61	21.64		0.89	0.00	0.89	0.89
wraps:	38.55	0.00	38.55	38.56		9.05	0.00	9.05	9.05
skips:	51.53	0.02	51.55	51.69		0.34	0.00	0.34	0.34

The file containing the input lists in yaml occupied 678 KB on disk and I had 2+ meg of RAM available as I started this run. I don’t think these results can be explained by memory problems. eg using virtual RAM on disk. It appears that the Enumerator based approach is much slower than the approach you used! I thought my suggestion was much easier to read and understand but the apparent performance difference has to make your code the clear winner. ;-)

btw. I also ran this with far larger lists that clearly exceeded available memory and found that the Enumerator based approach appeared faster. I think that may be because that approach is not supposed to require as much memory so perhaps there was less HD thrashing going on.

The take home message is of course:

When performance matters, test your code for performance.

I can send you the code for my tests, if you would like.

If you have two lists of the same size n that are equal, then you need Ω(n) time just to output the result, so the naive merging algorithm is optimal. In other words, you may only expect to do better if (1) you look at the average-case instead of the worst-case or (2) lists have significantly different sizes. The second point leads to the idea of iterating through the short list and checking for each element if it is in the other list. Checking if an integer is in a set is done in expected constant time with hash-tables. You might want to try it.

Because hash-tables are not terribly cache-friendly (which reminds me, use linear probing not chaining if you want it fast!) you could install a Bloom filter in front of it that quickly discards many of the elements that aren’t in the set. This is probably worth it only for the big lists/hashtables.

This brings the worst-case down from O(m+n) to O(min(m,n)). Analyzing at the average case is more tricky.

The other thing you can do is to wrap the code in a “pre” tag e.g.:

def hash
   title.hash
end

but I believe it would still be better to put spaces in the spaceship operator just to make sure.

I’ve fixed up your comment for the moment. If it does happen again despite all this, I’ll have a look at fixing it in a more fundamental way.

This is awesome, I’ll need to spend a little bit of time playing around with your code before I can say anything intelligent so bear with me :). It might even end up as a separate post :).

I hope you continue to find my blog interesting.

Is there anything I can do to avoid similar problems in the future?

module SortedListOps
# This is the method for a 'raw' list (a sorted list of unique integers)
  def isect_raw(list1,list2)
  	result = []
  	e1, e2 = list1.each, list2.each
  	a, b = e1.next, e2.next
  	while true
  	  begin
  		case a < = > b
  		  when -1 then a = e1.next
  		  when +1 then b = e2.next
  		  else result << a ;a = e1.next; b = e2.next
  		end
  	  rescue StopIteration; return result
  	  end
  	end
  end
# This does the same for a list of integers  now wrapped in ostructs.
  def isect_wrapped(list1,list2)
  	result = []
  	e1, e2 = list1.each, list2.each
  	a, b = e1.next, b = e2.next
  	loop do
  	  begin
  		case a.value < = > b.value
  		  when -1 then a = e1.next
  		  when +1 then b = e2.next
  		  else result << a.value ;a = e1.next; b = e2.next
  		end
  	  rescue StopIteration; return result
  	  end
  	end
  end
# Finally the intersection method for sorted ostructs containing an integer value and a skip_value representing the value of the ostruct one skip_size further along in the list.
  def isect_wrapped_with_skips(list1,list2)
  	result = []
  	skip_size1 = Math.sqrt(list1.size).to_i
  	skip_size2 = Math.sqrt(list2.size).to_i
  	s = 1
  	e1 = list1.enum_for(:skip, s)
  	e2 = list2.enum_for(:skip, s)
  	a, b = e1.next, b = e2.next
  	loop do
  	  begin
  		case a.value < = > b.value
  		  when -1 then s = (a.skip_value && (a.skip_value < b.value) ? skip_size1 : 1); a = e1.next
  		  when +1 then s = (b.skip_value && (b.skip_value< a.value) ? skip_size2 : 1); b = e2.next
  		  else result << a.value; s = 1; a = e1.next; b = e2.next
  		end
  	  rescue StopIteration; return result
  	  end
  	end
  end
end
class Array
  def skip(s = 1)
  	current_index = -1
  	loop do
  	  current_index += s
  	  raise StopIteration unless current_index < self.length
  	  yield self[current_index] || self.last
  	end
  end
end

The isect_wrapped_with_skips method depends on defining a skip iterator for Array. I take advantage of the fact that the arguments passed to an enumerator created by enumerable.enum_for(:method, *args) are available to the enumerator where ever it is used. Assignments to those args are seen within the enumerator and thus can be used to affect the enumerator behavior. Note also that you cannot use enumerator.first to set the initial values. This does not set the position within the enumerable in a way to support the correct behavior of the first use of enumerator.next. This seems to me to be a bug but I'll have to investigate further.

When I put these routines through your timing routines they appear to be quite fast but I haven't had time to compare them directly with your versions.

I'm very glad to have found your blog. It has been very stimulating. Thanks!

btw. My reply to your post (http://www.skorks.com/2010/03/writing-a-more-ruby-ish-array-intersection-function-and-sorting-structs/) contained several errors. I wrote that code off the top of my head and failed to test it at all before submitting it. :-( The current code does work in general but there are edge cases that need to be tested before I would use in any production code.