"Fixing Rflickr and Xmlrpc to Handle Bignums"
Jan 2, 2008 3:24pm |
3 comments
Flickr has become one of the more popular online photo storage sites and consequently has billions of picture on there. I use my Flickr account, in conjunction with a modded rflickr gem, to display all my photos on my website. It uses the Flickr API to get the pictures by ID, the downloads and caches the pics. Lets talk about these Flickr photos and their identifying ID's. Every time a new pic is added, it is assigned a numerical ID number, and that ID number has been counting up since day one. In fact, there are so many pics, and the IDs are so big that the ID number of the pics can no longer fit into an Integer container, but now need a Bignum or Bigint to hold the value. Let me explain... for example, one of my more recent pics:
... has an id number of 2153550206. This number in hex is 0x805C917E but more importantly, in binary it is:
1 2 3 |
Decimal: 2153550206 Hex: 0x805C917E Binary: 1000 0000 0101 1100 1001 0001 0111 1110 |
"So what" you ask? Let me talk a little bit about signed integers. All modern computers use a binary counting method called two's compliment to create a signed integer, as in, the integer has a negative sign or a positive sign. In this system, the first bit determines the sign, 0 for positive and 1 for negative. This limits the value of the most positive int to be a 0, followed by 31 1's, A.K.A. 2^31-1. This maximum value is 2147483647. Uh-oh... The number id of my picture is bigger than that... how can I represent this value using an int? The answer (in ruby) is using a Bignum which is simply twice as long (64 bits) so that there is plenty of room to spare. For example, my number above would become:
1 2 3 |
Decimal: 2153550206 Hex: 0x00000000 0x805C917E Binary: 0000 0000 0000 0000 0000 0000 0000 0000 1000 0000 0101 1100 1001 0001 0111 1110 |
... Giving us 2^63-1 positive numbers instead of 2^31-1. Luckily for us, most of ruby detects and switches to the Bignum seamlessly. That cannot be said for all of the ruby libraries though. I recently came across a problem in which this large picture ID was throwing up an exception because it could not be contained in a 32-bit integer container:
Bignum is too big! Must be signed 32-bit integer! |
Tracing the error stack, I was able to look into where the error was generated... /opt/local/lib/ruby/1.8/xmlrpc/create.rb
1 2 3 4 5 6 7 8 9 10 |
when Bignum if Config::ENABLE_BIGINT @writer.tag("i4", param.to_s) else if param >= -(2**31) and param <= (2**31-1) @writer.tag("i4", param.to_s) else raise "Bignum is too big! Must be signed 32-bit integer!" end end |
It seems that for some reason, the xmlrpc library doesn't switch over by itself... I can either 1) ask Flickr to change how they represent a photo, 2) rewrite a bunch of code to handle the photos in a different way, or 3) I must set this "Config::ENABLE_BIGINT" value to be true. Lucky for us, option 3 is a one-line fix over in /opt/local/lib/ruby/1.8/xmlrpc/config.rb, line 26:
Change this:
ENABLE_BIGINT = false |
... To this:
ENABLE_BIGINT = true |
Now save and restart your server and all should work. This should work fine if you are on your own server or running locally, but if you are on Dreamhost or another shared server, you must first install your own copy of ruby to make the change to as you do not have permission to change the servers copy of ruby. If you use this method, your file to change will be located somewhere different than /opt/local/lib/ and you will have to do a:
find . -name config.rb |
... from your home directory to find the file and make the change. After following all the instructions, my file was located at ~/run/lib/ruby/1.8/xmlrpc/config.rb Good Luck!
EDIT: Just for fun... the last pic whose ID fits into an integer (ID = 2147483647) is:
And the first Bigint pic (ID = 2147483648) is strangely not found...
The requested Flickr photo was not found.
Photo ID: 2147483648
Comments
Thanks for your tip. I ran into the same problem and am trying to fix it. I installed ruby on Dreamhost without a problem and changed the config file, but unfortunately no dice.
Do I need to tell Rails to use my version of ruby now? I tried changing the bang line in the dispatch.* files, but after restarting the server, rails failed to start.
Any tips? Thanks in advance.
First you have to edit your $PATH variable, set in either ~/.bashrc or ~/.bash_profile. You must prepend your path with the path to your local copy of ruby. After you save it, enter "source .bash_profile" (or bashrc)
Next, verify that your copy will be used by typing "which ruby". You should see the location where you have your custom install.
Finally, in your public/dispatch.fcgi file, change the top line to be "#!/PATH/RETURNED/BY/STEP/2"
Hope this helps... if not, email me with your exact error through the contact form here
dude. just solved this way easier for those of us on a shared host. check it. all you have to do is override XMLRPC. which rflickr loads from base.rb.
1. copy your local ruby's version of xmlrpc to your rails lib/ folder.
2. edit /lib/xmlrpc/config.rb, line 26 to "ENABLE_BIGINT = true"
3. done! your local /lib/ version of xmlrpc gets loaded instead of the standard ruby lib. booyeah.