knowledge For All: January 2015

HTTP Responce Codes

When we're writing a mobile client or a web based client it's going to access resources on a server. One of the things that we need to understand is how to interpret the response codes that the server sends back to us. We have to know how to read what the server is telling us happened on the server from the numeric code. because this is a really important part of writing the logic in our client to appropriately react to what took place in the server.

We may need to re-send the request or re-try it, or we may need to go and look at our code because we see that there's an error in how we format it. So let's talk about how response codes are formatted and represented in the http protocol. So there's a series of response codes that are typically in the range of 100 to into the 500s. And the one XXX messages whence you will see for example. You might have 104, 105 anything in the hundred range is an informational message.

So the server's telling us something. The quest isn't completely processed yet. But it's giving us some information about your request. Now the next one that we care about are things that were successful. So request codes that start with two are successful request codes. So you asked for a resource, I found it, okay, I'm giving it to you. So one of the really important ones that we want to know from the two series is 200 which is okay. And this is basically the server telling you, you sent me a request, I found the resource, I took whatever action you wanted on that particular resource. It's okay, everything worked.

And the data that you're getting back in the body you can go and interpret it and you can assume that things were successful. This is a very important status code to know. Now, one thing to remember when you're building clients is you should remember the scheme that you used. I'm talking about 1XX, 2XX. It's important that when you're building your logic you be able to quickly differentiate between something that's successful verses something that's unsuccessful as I'll show in a second. 3XX is typically redirection. It means that, for example, something has moved. The client is going to have to do some extra work in order to completely send that request and get it processed.

So, for example, you know, you asked for a request. Are you sent a request on a resource that is now moved somewhere else. And so the server's going to tell you the new location of that resource and you are required to resend your request in order to have it processed and completed. So this is some type of redirection typically. Further work that the client's going to have to do in order to get its request completed. So it's not necessarily that the request was unsuccessful it was just that we have to do more. We're probably going to have to resend the request to a different location that the server's going to give us. That's typically what's happening with the three series.

The next series is 4XX. And what this is, is you messed up. The client messed up. So the server's going to tell you, that either the resource you requested doesn't exist. So for example 404 is a common error code that you're probably going to want to handle. And that's that the resource wasn't found or you may have, depending on if you're sending parameters to the data, you may have a response telling you that you've formatted your requests incorrectly.

So, for an example, in later parts of the course, we'll see when we're building in our web-based services and applications to handle data from the clients. We actually can enforce the rules that certain data is set from the request. And if data that we expect to be there is not, we can send back a response code to the client, telling it that its request was formatted improperly.

And then the next series of response codes that we want to be aware of are the five series. And what these are are server errors. So this is something bad went wrong on the server. So a common one that you'll see is error 500 which is some error took place on the server you as the client need to know about because we didn't successfully complete your request and often the body of the message might tell you information about the error. So, for example, if you're building a or communication with a Java-based application, the application may return a stack trace to you in the body of the message telling you exactly what went wrong.

Although, you typically don't want to do that. You don't want to send to much information like a stack trace back in your response to a client because you don't want them to know that information. For security purposes you shouldn't be doing that. But it is helpful to have information about what went wrong to some degree. So when your building a client you want to build logic that reads these response codes and then takes the appropriate action.

So you want to handle the response differently based on the code. So you don't want to go and send a request and then blindly assume that it was correct, read the data out of the request and for example interpret that data as an image. You want to read the response code, see if you've gotten for example 200 okay. If you have you can go and interpret that data as an image. If you've gotten 5XX you shouldn't interpret that data as an image. You need to interpret as, you know, some information related to the error that, that server sent back to you. So you need to understand these response codes for you clients in order to put the right logic in them to evaluate the response that's coming back and determine if it was successfully completed or not.

HTTP Responce Codes

It's really important to understand the details of the HTTP protocol. In order to build and debug effective cloud services. So let's talk about those details in some more depth. So the HTTP protocol is always organized around a client sending a request to the server. So we have our server and our client and we have a request that's being sent.

Now, one of the key parts of that request is the action that the client is asking the server to take on its behalf. And so every request has a request method. And the request method, is the action or the verb that the client is asking the server to take on its behalf. And all requests are dictated as a request method that should be applied to a specific resource on the server. So for example, when you go and access a webpage, using your browser. What you're typically doing is sending a request that has the get request method. So it is a get request, and the resource is usually some webpage like index.html, which is usually the. Core webpage at a website, when you go to that, that address.

So the get is the request method, and the resource is the index.html. And the resource is typically specified as

a path to a resource on the server. So the resource will typically be a path. So you'll see something like get, you know, /index.html or fu/mypage or some other resource that you would like to access. And here again it's a Request method, and then a path to a resource. So let's talk about the H request methods. There's a variety of them in the protocol, but there's a subset of them that we really care about for communicating between mobile devices and the cloud.

One of the most important ones that we want to understand is Get. Get is a simple request to the server it can be without data or possibly include a little bit of data and it's asking the server to get some resource that's there and return it to us. So we're going to talk about the important HTTP request methods, that we really care about.

Another really one, important one is POST. POST is typically used when you want to send a lot of data to the server. for example, if you want to go and Post an image to the server that it can then store and serve up at some later point in time. Post is probably something you will be using to do that. Get you probably aren't going to be sending an image through get. You're going to be sending some small amount of data through Get. Post is your more general purpose send data to the server. These are your two most important, but there's two more that's worth knowing about having some familiarity with.

Another one is Put is asking a server to store some data that is contained within the request on the server. And the last one you probably want to know about is Delete. And it's fairly self explanatory. We weren't asking the server to delete some information on the server. So when you're sending HTTP requests, you're always going to specify an HTTP request method plus a path or resource. That you want that method to be applied to you.

Whenever a client sends a request to a server using HTTP, that request has to have a very specific format. I mean, just think about it, if you're that server receiving all of these requests from all over the place. You have to absolutely make sure that they all have the same format so you can interpret them properly and figure out the right thing to do with them. And that's why HTTP, very strictly specifies what the format and the rules are for sending that request.

So we have our HTTP Request. And, each HTTP Request has a number of critical, some mandatory and some optional, parts that have to be sent to the server. One of the first parts that has to be sent to the server is the Request Line. And the Request Line specifies two really important things. The first thing is it specifies the Request Method. And the second thing that it specifies is the Resource.

That we want to take action on, which is typically specified as a Path. So these are a request part. Now, this is the key thing. We're going to say, server, please go and take this action. On this particular Resource. But when that server gets that Request there's lots of cases where it may have multiple options for how it goes and completes that request. For example, let's say that we have a webpage on the server, and it's the homepage of that particular web address, so you go to the Equivalent of google.com. Well, sometimes the server needs help knowing some information about the client that can help it to complete that request, things like the language that the client would like to receive the response in.

the next piece of the request is a series of Headers. And what we can think of these is extra information to help the server. And these were things like the language that we would like the response to come back in. Or the character set that we would like to see the response in, or the content type, what we're sending back to the client so it knows how to interpret it and process it. Or possibly something like the cookies or small pieces of data that were sent from the server to the client in a past request that the client's now providing back to the server to help it figure out. Where it was or who this person is, and associate them with past requests.

the Headers are the second key component of a request, and they provide extra information to help the server figure out the right way to process the request. A final piece of information that can be part of an HTTP Request. Is the Request Body, and the Body is any data that the client is sending to the server, in order to help it complete the request. Now, this an optional part of a request, and not a 100% required, but let's distinguish it a little bit from the Headers. The Headers are meta information, things to help it know the right way to process the request.

The Body is core data that is being sent to the server to process the request. Now one way to think about this is in most cases. If you didn't include the Headers, but you did include the Body, the server could still process the Request. It just may not give it back to you in the format that you expected or exactly the way that you expected it. But if you didn't include the Body and the server needed the Body, it wouldn't be able to process the Request. So the Body is the data the client is sending, that the server absolutely has to have in order to complete that Request. The Headers are extra information that the client is giving the server to help it complete that Request.

Every time we're sending a request to the server, we're asking it to take some action on some resource that we're interested in. So, obviously, a really important part of understanding how to write clients that operate on HTTP is understanding how we talk about resources to the server and how we identify the resources that we're interested in so that the server can locate them and then take that appropriate action. Now in HTTP, the way that resources are identified is with what's called a uniform resource locator or URL, which is what everybody should be familiar with. And what a URL consists of is the specification of the protocol, which in this case, is http://, a host or server that's going to be talked to. Potentially a port number that's preceded by a colon and that is, we're going to connect to a specific port on that host. And then a path, that we would like to access, representing that resource.

So, in this case, this is telling us which server to go and talk to on the network. Which port of that server we want to deliver our HTTP messages to. So this is typically going to be the port that the web application container or web server, whatever the entity is that's providing the server side of the HTTP communications. This is going to be the port on this host that it's listing. And then finally you have the path to the resource that we are interested in on that particular host.

So then when the host receives this, what it's really interested in is typically the component at the end; which is the path. It already knows where it is, it already knows its port, but it needs to know the path to the resource that you're looking for. One other interesting component of URL's that are very important in HTTP communication from mobile clients and other clients trying to communicate with web based applications is the concept of query parameters. So you may have seen some URL's that look something like this. And what this is, is at the end of this we have what are called query parameters. And what query parameters are, is they are additional information that can be attached to the end of the URL, that's passed along to the server in order to communicate information, additional information, about what particular aspect of that resource the client is interested in. And these query parameters take the form of, they must follow a question mark that is placed at the end of the core part of the URL.

So right after the resource path, we have a question mark, which we see right here. And then we have a series of key value pairs, where in this case we have a key which is A and then we have the value which is B.

And so these query parameters follow at the end of the path that's being specified. Now if we want to specify multiple query parameters, that's also possible too. In this case we only have a single key value pair. We have the key A equals B. We can also pass multiple parameters. So for example, if we had a question mark and then we had A equals B, we can then add an ampersand to the query path, I mean the query parameters. And then we can have a second set of key valued pairs. So, in this case we have the first key is

key 1 which is A. Then, the second key which is key 2, which is C. And then we would have B as the first value for key 1. And D is the value for key 2.

So using query parameters, we can pass essentially extra information about a specific aspect of a resource to

the server. And we can pass as many of these as we want up to a limit. Now, we just keep separating these query parameters with ampersand in order to understand where the separations are between the keys and the values. There are a couple of important concepts that we need to know about, if were going to pass query parameters to the client. One of those things is we can't have certain special characters appear within our query parameter values and keys. So as you can imagine on the server side, the server needs to easily be able to parse these query parameters by looking for the equal signs and the ampersand signs as well as the question mark.

And if we began inserting these special symbols into our query parameters, it'd be hard for the server to extract them out. So, by default there are a set of characters that we're allowed to have in our query parameters and a set of characters that we aren't allowed to have in our query parameters. And any of those characters that aren't allowed to be in our query parameters have to be actually encoded using a process that's called URL encoding. And the idea behind this is, if we want to pass a value that has characters that aren't allowed in the query parameter spec.

What we can do is instead replace those characters, with what are called their URL encoded equivalence so that we can still represent them, and the server understands how we represent them. But when we actually go and look at the characters there itself, none of these symbols show up. They get replaced with other things that mean the same things. So if we have an equals sign within our value for a particular key pair, it will get replaced with the URL encoded equivalent of equal sign. Now, you could go and look up all of the specific ways that you URL encode different characters into a query parameter, but you don't need to. Almost every programming language that you're going to work with has libraries and functions available to you to automatically encode your query parameters into a properly URL encoded string.

So you don't need to go and learn them, you just need to understand the concept that. If you pass specific things like the equal sign or the ampersand or the question mark or the slash or space inside of one of your keys or values, then you need to make sure that you're URL is URL encoded. And so other than that, you don't need to do anything. But you do need to understand the cases where if you put something into one of your query parameters, you need to URL encode it. Often, when we're writing clients, they're going to be sending data to the server or sending requests to the server, we want to go ahead and URL encode all of our URLs that are dynamically constructed with some type of data that we're taking in. Because there's a possibility that that data may not strictly adhere to the query parameter specification.

How Hypertext Transfer Protocol (HTTP) Works

So there's different estimates, but people estimate that Amazon has between 150,000 and 450,000 servers that are backing its public cloud that you can get access to. Similarly, there are estimates that Google and Microsoft have over a million different servers that are backing their infrastructure that they use to support things like Gmail, or Google search or Bing search.

So cloud services have become really a fundamental capability that you need to be able to build really large scale and compelling applications, like Amazon and Google and Microsoft do, or the many start-ups that are doing really fascinating things.

Now there's lots of material on building web based services. using protocols like the Hypertext Transfer Protocol, or HTTP. So what do we need if we're going to talk to a server that's sitting remote from our mobile device?

So we've got our mobile device that's sitting here, and we're going to communicate between it and some cloud resources. So what do we need to do in order to support that interaction? Well, one of the most important things we have to do is we have to select a protocol for talking to that cloud. How do we go in about and talk to the cloud, the cloud, and that's what a protocol is.

A protocol is a set of rules dictating how we communicate with the resources on the other end of this network. Now why do we need a protocol, or some specialized way of communicating, and what does the
protocol get us? So if we just go and make a local method call, this is going to be, you know, very fast, nearly instantaneous.

But when we go and do some communication over the network, for example, we communicate over a cellular connection to resources that are sitting in a cloud that could be, you know, thousands of miles away, this could be milliseconds to seconds to do that communication. So there's a substantial longer period of time to establish that connection to transfer data and to receive some type of response that we're waiting for. So a big question that comes up is, well, how long is too long? When should I disconnect from that server and assume that my communication from my mobile device failed?

because, we've all experienced it, where you're driving along and suddenly your call drops because your signal goes away. Well how does that cloud know when it's been too long since it's heard something from you? Or, what if you go and log in to a website, when should it, decide to De-authenticate you and make you log in again? All of these things are aspects of the protocols that we design in order to communicate with some other entity that's farther away. And it's really important to have a robust protocol for this communication to ensure that we interact securely and reliably and quickly with the resources that we're trying to interact with.

So to give you an example, think about the Mars Rover that, that NASA has launched and been controlling from the earth. When they send a message from Earth to the Mars Rover to drive it or control it, it can take a substantial period of time for that message to receive, to be received on Mars by that rover. It's not milliseconds or seconds on the, it's on the order of hours. So think about it, you're driving this rover, and suddenly it hasn't received a message in an hour. What should it do? How should it respond? Should it keep going at 50 miles an hour or 10 miles an hour or however fast the rover goes, or should it stop? Should it try to return to where it started from?

So this is what a protocol defines, is it tells you how you communicate with another entity. And it defines a syntax, or basically a format, for the messages that you can send to that other entity. So it says, when I send a message, it has to have these properties to it, and that's the syntax. It defines semantics, and this is basically the meaning of the messages. So if I send you message A and then I send you message B, here's what I mean by that, I don't want you to get confused, I want you to understand the semantics of my messages, or the meaning of them. And then it spells out the timing, so I if I send you message A, how long do I have before I should send you message B? Or should I even send you message B at all? Should I send you something else instead? So building and selecting the right protocol for the interaction with the cloud is one of the most important aspects of mobile to cloud interactions.

So how do we talk to 150,000 servers in Amazon EC2? How does a lonely little mobile device, that's connected to some network, out in the wild. Get access to these thousands of servers, that are sitting in some cloud computing environment. So how do we get from there to there? Well the way that we're going to talk about getting from the mobile device to the cloud is through the use of HTTP. And what you can think of HTTP is, it's basically a dialog or a language that a mobile device can use to speak to a cloud, it's a protocol, an application protocol, for communicating between devices on a network.

Now. We're going to use HTTP here as our protocol for communication. But it's helpful to have a little bit of
history. And understand what HTTP, was originally built for. HTTP is the protocol, or the way that your browser communicates with a web server. In order to retrieve resources from that web server such as a web page.

So when you go and you log in to Facebook, for example, your browser is sending HTTP, probably a series of HTTP requests, to one or more Facebook servers. And then that server, or servers that are receiving the request or requests. Send data back to your browser. HTTP is a client-server protocol and what that means
is that, the interactions that are mediated via the HTTP protocol, are always initiated by a client that wants to.
Access or request, some resource on the server.

So the server has a series of resources, that it can provide to the client, and the client sends requests for those resources and the server. Returns responses to those requests. And the responses may have the resource that was requested or it may have a message indicating why the Server could or couldn't process the request, that the client sent.

But fundamentally, the HTTP protocol is an application protocol that is client-server. So, the client will always be initiating the communications. And there are some changes to this with newer things like web sockets. But in general, the client will always start the interaction. And the server will receive the interactions. And process them. So, this is the fundamental way, that HTTP works.

Now the client can be either a mobile device as we'll focus on in this class, or it can be a browser. So, we can have a mobile device be the client, or we can have a browser be the client. Or we could have some other device, maybe in the internet of things, we use HTTP as the protocol to communicate between different pieces of equipment in a factory and the different controllers, that are monitoring their status. So HTTP is a general purpose protocol. That can be used, to communicate between a client and a server.

Now one of the reasons that HTTP has become so popular for mobile integration with cloud, is that HTTP is a widely used protocol already. All of the websites that you visit are already using. HTTP as the protocol that's being used to access those web res, resources. So there's already so much infrastructure that's been built up and so much investment that has been put into HTTP protocol, and so much understanding of how the HTTP protocol works, that organizations naturally want to reuse. That infrastructure and investment, on their mobile platforms. So by using HTTP, they can write a server side, that provides access to these resources, which could be a service, or could be a web page. And, they can provide a common interface
to those resources, through HTTP.

It's a common protocol and a common server site interface, to access those resources. So when a server sends a response back, it doesn't necessarily have to care whether the person or thing that's consuming that response is a mobile device or a browser, or some other thing that it's never seen before. As long as that thing knows how to speak the HTTP protocol, it can consume the resources and services, that are available on that server.

So why are we using a web browser protocol to integrate mobile clients with cloud services? Why do we want to use HTTP to talk to the cloud? Well HTTP provides us a number of nice properties. One is, we've got a uniform interface that we can provide via HTTP to the services or resources that are sitting on the server. And, for example, if a client is a Browser and sends a request over HTTP and that Server rec, receives that request, the request is exactly the same as if a mobile device had sent it.

So, if this is a Mobile device that's sending that request, either way on the other side, on the server side, that request still has to conform to the same requirements and specifications of the HTTP protocol. So we can have a common interface. This is one of the big advantages of using HTTP to interact with cloud resources and services.

Is we don't have to go and rewrite everything from scratch and figure out a whole new way of having our servers accessible and having mobile devices able to communicate with them. We can reuse what we're already building and providing to our browser based clients and the web is ubiquitous. Everybody's building websites, everybody's supporting web service as it is.Why not reuse this infrastructure for mobile? And so that's one of the big reasons behind using HTTP to support mobile and cloud interactions.

Now, we get a number or of other great things as well. There's already all kinds of stuff on the server side from a framework and programmatic perspective that we can use to process HTTP requests. So there's things for doing stuff like sessions or remembering when people request something, who that person is and what they previously requested or what other data is associated with their current login.

We also get things like Data marshaling. This is very important when you want to convert from the format of the HTTP request to some other format that you need to process. And there's a huge amount of infrastructure that's already been developed, in terms of libraries and frameworks, that you can use to program if you use HTTP as your underlying protocol for your interaction with the cloud.

But there's also a number of other reasons that it's really nice to use HTTP. There's also infrastructure that decides, how do we load balance and distribute requests across our servers? So for example, if you have a million people accessing your service simultaneously, how are you going to distribute those requests across those servers? So, Load Balancing infrastructure for HTTP can be reused if we're using HTTP as the communication mechanism with the cloud. So there's such a significant investment both in the existing frameworks and libraries, the load balancing and other management infrastructure that's been developed for HTTP. It's natural to use HTTP as an interaction mechanism with the cloud.

Now, there's some challenges to using HTTP, particularly when things that we would like to do with our mobile services don't fit into the client server model where the client is always the mobile device requesting some resource from the server. So maybe an email comes into the server and your client needs to be notified about it. Well that's difficult to fit into the HTTP model. But we'll talk about how we overcome some of those challenges later.

So what is exactly is a cloud service, that a mobile device might talk to, and how do we differentiate and understand, how these services differ from the typical applications. That we're used to writing. So, let's take a look at an example and hopefully it'll help to clarify and better understand what a cloud service is, and what the scope and what are the challenges that we're going to face, when building these services.

So let's assume we're writing a Java application. And what that java application allows us to do, is we can capture a video and then we can run a command line utility that will be our application. And it will go in save the video, on the underline file system. And not only will be going and save the video in the underlying file system, but it will also take that video and it will save it to a database. So, that we have a record of all of the videos, that we've entered into our system.

So, this is our service in this case or application. That we're going to be building. And we'll assume that there's nothing special about this, it's just a Java application. Well, if we're going to be running this thing over here from the command line, one of the things that we're probably going to want to do, if we're writing this in Java is we're goig to have a main method. So, we might have a public, static, void, main that takes some arguments. And we could go out and spell out what those are. And then in that main method. We might go and do things like you know, look for the flags that were passed on the command line.

So, we can look for flags. We can look for the file that they want to add the video. So, look for the video path, and we can do a whole bunch of other stuff. But the key would be, we've got this main method, which is part of this application and every time from the command line they invoke our application. They're going to be passing commands from the command line into our application, so what we have is a stream of commands that are being passed into our application. So, every time the user goes and invokes this thing from the command line, they're basically sending a different command to it. And the tax that they enter in the command line is then being passed to our public static void main method, where we can start up our application, take action based on the data that they passed us, then going store something in a database, or do something else.

We might also want to issue commands to our application. That have it go and query the information we've already stored, or be processed based on the information we've already set the application. So one of the, the things we might have to do then, is provide feedback to the user that's sending the commands to the application. So we might have something in our main method that's printing out feedback. Using System.out.printline back to the user, so that the user can see the results of the commands that they're issuing. Or see information that comes out of the database that the user is querying. So, not only do we have a situation, where we are sending commands and asking. The application to go and store data, but we also have situations where we need to, based on those commands, go and provide feedback to the user, about the success or failure of those commands and any data associated with them.

So, for example, if the user issues a query to see all of the videos created. In a certain time period we'll probably want to print out the query back to the command line. Similarly if the user issues a query that's malformed. We would probably expect the application to go in print on the command line an error message. Stating what the user did wrong, and helping them to figure out how to formulate that command correctly and send it.

Where if the application went and processed the command, and it had a bug in the program and it crashed or we would expect to get some feedback telling us that the application crashed. So what we'll see is this basic architecture of commands being sent to an application processing data in some format either storing it or performing some analysis on it, and then providing feedback to the user based on the success or failure of those commands, and in the results of the execution of those commands.

The big difference is when we go into the cloud services model, what we're going to have is, the case where we're sending the commands to the application from a mobile device, that's not sitting on the same app, the same host as that application. So this mobile device is going to be sending this command remotely over the internet. To this application and this application's going to be living in the cloud and receiving those commands, processing data through some data store, saving the information somewhere that's being sent to it, and having to remotely send back to the mobile device, the results of those commands. So, we have this remote interactions that's based on commands being issued to the application and results coming back, that we're going to be working with.

We're going to be sending data from the mobile device, which is the client, to the application. It's going to process that information. And in this case, the application is going to be a server. And it's going to be living in the cloud and then it's going to send the results back to the mobile device, so that it can know what happened. So it may be sending back results that say here's the result of running your command. It may send results back that say you know, did not formulate your command correctly. Or it may be sending results back saying the server had a problem.

Here's what the problem was and it crashed. And that's the architecture that we're going to be building through this class. Is this remote distributed architecture where mobile clients, your mobile devices on Android, are sending commands over the internet, to a service or a server that's living in the cloud. And they're being processed and then results are being sent back to the mobile client. And one of the key questions that we have to answer and address is, how do we get these commands from the mobile device to the server and communicate the results back? What are the requirements and rules for that communication with they server that's remote? How do we ensure that it's reliable? How do we ensure that it's orderly, and done in a way that the server can understand and process commands for us?