Learn Concurrency and Socket Programming with Go by implementing a made-up protocol called Short Text Transfer Protocol (STTP). Although Go is the featured language, the project can be implemented in any language you desire.
Imagine a computer network as a postal system with multiple layers. Just like the post office delivers letters, a network delivers data between computers. Each computer has an "address" (IP address), ensuring data goes to the right place. Data is packaged in "packets," with each layer adding information like an envelope, stamp, or address label (encapsulation). Routers, like postal workers, use this info to guide packets to their correct destinations through the network's layers.
Tip: When talking about networks we refer to a model called the OSI model and the layer we're working with or having troubleshooting.
The client server model is the base architecture for how one device or software application makes a request and receives a response from another device or application.
The OSI model is a conceptual model used by engineers to talk about computer networks. Software and devices each play a role within the model interacting at different "layers".
Watch videos 1-5 of this playlist. If you were trying to become a network engineer this is super important, but as a software engineer, just watch and absorb the information.
A reliable protocol that ensures data is delivered accurately and in order between devices over a network, like a secure, tracked delivery service.
Translates human-readable domain names (e.g., example.com) into IP addresses, acting like the internet's phonebook to route requests to the correct server.
The following link includes a table breaking down the OSI Model along with which tools you can use to interact and troubleshoot each layer.
Chris Greer is one of the best presenters of TCP/IP content I've come across, linked is one of his talks. If you ever need to really dive deep, or are curious, hard to beat his content.
A socket is an endpoint for network communication between two machines or processes, defined by an IP address and port number. It enables data exchange over networks using protocols like TCP or UDP. Socket programming involves creating, managing, and using sockets to build networked applications, such as web servers or chat clients, where data is sent and received between connected systems. This section will introduce you to programming with sockets.
Tip: When working with socket or network packages, they can be a bit overwhelming to work with given all the options and toggles available to any one protocol. I tend to stick to examples and tutorials as I find I need to see a complete working example to understand the big picture.
This is a great self contained example of building a simple client and server application in Go.
Sockets and buffers are key to socket programming.
You'll implement a fake protocol over TCP called Short Text Transfer Protocol (STTP). Up to and including "(F) Error Handling and Response".
If you're interested in the Python example, here it is. I recommend sticking with Go for network programming for any serious project though.
You can also implement it in Python as well. If you're having trouble switching between languages, stick with Go if you can.
Caching is the process of storing frequently accessed data in a fast, temporary storage layer (in memory or caching service) to quickly retrieve it without repeatedly accessing the slower primary storage (filesystem or a Database). It improves performance by reducing latency, and speeding up data retrieval, which can enhance the responsiveness of your apps and services, especially under high demand. This section will introduce you to basic caching.
Tip: Caching falls into the knowledge category of "Systems Design". Which looks at the overall architecture of a software system as a whole to addresses problems of performance, latency, high availability (ensuring your apps do not become unavailable), and more. It's a whole discipline in its own right, this is just scratching the surface.
Engineers use performance testing tools to understand how well their applications are performing under high load. These tools can be expensive tooling, or simple Bash scripts.
Play with the provided script until your program breaks under load. Make note of the errors you receive.
Complete the stories under the "(F) Caching System" section. Run the performance script throughout the development process to see improvement.
This is the really fun stuff and where Go really shines. Concurrency is allows your program to do multiple things at the same time. In other languages it's common to refer to concurrency as asynchronous programming. In this section we're going to learn a bit about concurrency and then implement it within the STTP server.
Tip: There are a lot of new concepts and terms when working with concurrency, just take it slow and work on understanding once concept at a time.
Read the following articles to get an idea of what concurrency is about. Follow the examples yourself and play with their behavior.
This is a great series and goes deeper into concurrency. Dedicate the time to watch all videos. The presenter whiteboards along the way with examples.
Included here are other utilities and features to be aware of when dealing with concurrency.
Complete the stories under the "(F) Concurrency and Thread Safety" section. Ensure the performance script is making concurrent requests and run the script throughout the development process to see improvement.
This is the last section dealing with socket programming but this is just the surface. We'll close out this guide with implementing the STTP Client and then provide some optional projects and ideas to go further if you desire.
Complete the final set of features starting with "(F) CLI -". This will implement a client side of the socket programming, along with introduce topics such as name resolution, url parsing etc.
You should be familiar with what the performance test utility does at this point, can you rewrite it using Go, Python or another language?
Other ideas to implement if you're looking to expand this project
Another advanced topic to cover is writing you own Proxies. This is pretty advanced stuff to implement, but fun challenge for experienced engineers.
Material found on the web can only go so far, and in this case for a very deep dive an expert understanding, refer to these books.