Another double-whammy post! I'll post in one entry two completely unrelated topics as to garner the maximum number of comment replies (which will probably be zero)...

First off, I think this is the year I've blossomed into a NBA fan. Now, they are showing the playoffs on TNT, and they have a commercial spot where they have the Black-Eyed Peas singing this jingle about 24 seconds (that's the shotclock in the NBA). The first time I saw that commercial, I remember thinking to myself, "Wow that is a really catchy song! I wish it wasn't just a jingle..."

As it turns out, it's a real song they adapted for that commercial! The song is "Let's Get Retarded" and it's off their latest, "Elephunk!" I was so happy to listen to this song...

Now for some technical stuff.

Most of you will notice that Tabulas is spread across multiple sites; we have "xml.tabulets.com", where your XML (FOAF, OPML, and RSS) files are stored, and all of you are on "aces.tabulas.com," which is the personal server that stores media/gallery files.

Now, to get stuff stored on these separate servers, there requires communication between the servers. For example, let's call the primary Tabulas server "A" and aces.tabulas.com (the place where your images are) Tabulas server "B". How exactly does your image, which get uploaded to server A get to server B?


When you upload your image to Tabulas (Server A), the image is stored temporarily in a temp folder. Once Tabulas figures out how you want to style your image, it grabs the temporarily filename, your username, and the styles you want to apply to the image and then packages that request and sends it as a command to Server B.

Now the problem remains: "How do you create commands across separate servers?" Well, I've created a very rudimentary API between the two servers where Server A sends data as a POST form to Server B. However, you cannot just allow any plaintext commands being sent to Server B be executed, because a l33t hax0r may figure out the command system. So then the natural reaction is to also send in the POST data a "password" that both servers know. But how do you transmit this password? You obviously cannot send it plaintext, or someone will just get the password.

So herein lies the problem of how do you authenticate data being sent to two points? There are a few answers:

a.) You can run the connection over SSL.
b.) You can use some of the HTTP authentication methods.
c.) You can hash a password to send to the server.

There are plenty of problems with each of these:
a.) Running and creating SSL connections are very very CPU intensive. When you're just sending a simple command to Server B ("grab this image off my server and place it in this user's gallery folder), you don't want to invest a lot into the authentication connection.

b.) HTTP authentication encodes, but it's not much better than sending plain-text. So this really doesn't solve any problems.

c.) Hashing your password (with MD5 or SHA1) will prevent the password from being known, but it doesn't protect from "replay attacks." Basically people can take the hash and submit it directly with the commands ... so that doesn't help.

So what do we do? Well, Borst and I had these problems for the Lightbox7 Uploader ... how do you transmit data between client and server securely over HTTP (don't create new protocols) without getting too complex?

First, we hashed the password with a timestamp, which was also sent. So basically the POST data being sent from Server A to Server B contains the following information:
- The command to be executed
- The MD5 hash of (password.timestamp)
- the timestamp
However, this still doesn't protect against replay attacks; someone who is sniffing the network could just send this data again with a slightly different command.

So we created a 'nonce.' This was a one-time string that could only be used once ever.

So now we're sending:
- The command to be executed
- The MD5 hash of (password.timestamp.nonce)
- Timestamp
- Nonce

When Server B receives this information, it first checks to see if the nonce being sent has ever been used before; if it has, then Server B ignores the request... otherwise it accepts it as a valid command.

Now, what if the same nonce is generated twice? That is something very likely. In the case of the LB7 uploader, we decided the risk would be so small (Borst actually wrote a test program that checked to see how often the same nonce was generated and it never happened once during the execution of his script). However, in the case of the Tabulas, we cannot just gracefully fail. If it fails, it MUST store the failed request.

You see, the separate servers are not "linked" in any way; if the image servers and the database servers go out of sync, there is no way to sync them. So the burden is on Server A to remember which queries have been sent out and remember if they received and executed successfully; if they weren't, it must remember them so it can upgrade its database accordingly (you don't want to fill up your gallery quota with broken images because Server B didn't receive the request from Server A).


In any case, I started tackling the API between the new Tabulas CP and the XML server (xml.tabulets.com). I'm revamping it because the XML server has a really really crappy API (it's not even really an API per se). I'm building in the nonce authentication (which it is missing right now) and working on making the whole XML server a bit more elegant. The current XML server was hacked together as I added features to Tabulas and the architecture reflects that.

You may of noticed a few days ago (actually I bet no one did) that your XML feeds got a lot "cleaner." I actually went through and reformatted your XML files; if you view the "source" of the XML file, you'll notice clean formatting; each node has its own line and stuff. And I also cleaned up the newline and return carriages that mySQL loves to put in when you run nl2br().