Both upstream and downstream can have multiple connections concurrently, at a high level this is accomplished using async calls. Each connection instantiates a connection chain, which is functionally a state machine where each state is a message in the protocol. A message framework provides the structure for sending and receiving. It consists of two classes send_message and recv_message that take a state object, either a send_state or recv_state, respectively. The state object handles preparing and processing the message by implementing the operations used by send_message and recv_message.

class send_state
{
    ...
    // send operations
    virtual void prepare(connection_ptr conn, chasm::buffer& msg)
            throw (std::bad_alloc);
    virtual recv_state_ptr next_state() const;
    ...
};

Although briefly more than one state per connection will exist, usually there will only be one at any given point in time. The ephemeral nature of the states is accomplished through pointers and callbacks. A state is instantiated as a shared pointer and then started, shortly after that an async call will be made that will unwind the stack and the calling object will be destroyed. Callbacks provide a mechanism for keeping objects alive. The state becomes the callback through the use of bind and the this pointer. Following is a snippet of code demonstrating the concepts used:

void v0 :: send_message :: start()
{
    ...
    m_state_obj->prepare(m_conn, m_msg);
    ...
    send();
}

void v0 :: send_message :: send() const
{
    boost::asio::async_write(*m_conn->_socket, boost::asio::buffer(m_msg),
                             boost::bind(&v0::send_message::goto_next_state,
                                         shared_from_this(),
                                         boost::asio::placeholders::error));
}

void v0 :: send_message :: goto_next_state(...) const
{
    ...
    v0::recv_state_ptr state_obj = m_state_obj->next_state();
    if (state_obj.get())
    {
        boost::shared_ptr<v0::recv_message>
            rm(new v0::recv_message(m_conn, state_obj));
        rm->start();
    }
    ...
}

...

-mfm

I would like to digress a bit and talk about the whole design of the CHASM peer-to-peer protocol, eventually leading to design specifics with code samples. The material in this post is for a technical document I am writing, so it may come off as dry and boring to most. For those whose native tongue is C or C++, the new direction of this thread will have its redeeming qualities in the end.

"...the very best designers produce structures that are faster, smaller, simpler, cleaner, and produced with less effort."

Frederick P. Brooks Jr., The Mythical Man-Month

I did not know the real project was not in the code but in the design. A program that is well designed is not merely an accident: it is an act of human thought that transforms an idea into a tangible structure. I do not consider myself a great designer. In fact, I am not sure if I would know great design if I saw it. My goal is to learn what great design is, to strive for this ideal that Frederick P. Brooks Jr. speaks of. Thankfully, I have help in achieving this goal.

Not being experienced in design, I have had to learn numerous design patterns, patterns I previously knew only by name. Learning these, however, has not been a magical solution by any means. With a plethora of possible designs and my having only an abstract understanding of each, this has only added to my inability to decide on any one design. In the end, I did settle on a primitive design that has proven a good jumping-off point.

So, the design of the CHASM peer-to-peer protocol is now going through its second iteration in code -- many more never made it past my white-board. It has been an invaluable learning experience to receive feedback on the design from my mentors Ben Boeckel and Rob Escriva. After much discussion with them, I will implement a new design to reduce redundancy and temporary-state variables. This will increase clarity and provide a cleaner design.

In Part II, I will discuss the design in greater detail and provide a code snippet.

-mfm

CHASM now officially has a mailing list! The lists are posted at lists.robescriva.com. We have the traditional low-volume chasm-announce and the always exciting, and my personal favorite, chasm-dev lists. We encourage all people interested in changing Linux mirroring, even those working on competing products, to join.

-mfm

The CHASM protocol is still a work-in-progress; however, the part of the protocol that pertains to peer-to-peer communication is fairly complete. All communication between nodes will be application level messages as described in the technical design document. Any feedback about the design is more than welcome; my email address can be found in my git tree and we do have an irc channel #chasmd, so feel free to stop in and start up a conversation.

For the implementation of the protocol, I will be using Boost.Asio and all communication will be done using asynchronous calls -- at least for this initial version. The upstream (server) node will be responsible for receiving and servicing connections. The downstream (client) node will be on a timer that periodically checks with the Pool-Manifest Oracle for manifest modifications, and, when necessary, will connect to one or more upstream nodes to update the collection. Both nodes will communicate with the Pool Manager (handles active blobs) to read and write blobs.

In closing, my git tree is here and soon more of my actual code will be pushed there. Again, for feedback on anything I mentioned or on coding issues that you find, please contact me.

-mfm

Hello Fedora Community! Thank you all for making the Fedora Summer of Code possible. I am excited to announce that I will be working on CHASM this summer with my two mentors Ben and Robert. They have developed a great solution to a well-known problem, and I am grateful for the opportunity to work with them.

In short, CHASM is a Cryptographic-Hash-Algorithm-Secured Mirroring solution, it provides a true peer-to-peer network for keeping mirrors in sync. rsync, the canonical tool used for mirroring, has many deficiencies as it pertains to mirroring large repositories. Existing solutions that solve problems inherent to rsnyc, have used a top-down approach, hence, continuing their reliance on rsync for low-level functions. CHASM has been designed specifically for mirroring from the bottom-up. It will provide a stand-alone solution as well as a modern foundation for other high-level tools to run on.

Here is my FSoC proposal which provides a brief insight into who I am, along with my responsibilities and projected timeline for the project. The CHASM site contains more information and has an Atom feed for those interested in following all developer posts.

-mfm

We've published our evaluation for Fedora Summer Coding here. Please let us know if you find errors in it or have questions. As it says in the document, we'd rather you ask questions than leave answers blank. Please keep answers brief as we will not be requiring that len(response) correlate with quality(response).

-Robert

An unfortunate drive failure has cost us our test instances and all archive history for Fedora, CentOS, and Ubuntu. I am partially to blame in this (bad luck is to blame for the other part). When ordering and setting up the drives, I ordered only three drives, and put them in JBOD. I should have scrounged around for a fourth drive to make it a RAID system (then we'd at least still have our data).

In March, I purchased several 1TB hard drives for use in the CHASM project (using my research stipend from the Rensselaer Center for Open Source). Since then, my backup workstation (currently a headless server) has been syncing Fedora and Ubuntu every six hours and making a snapshot of the results. Our goal is to have an accurate profile of the nature of the changes in each mirror archive so that we may decrease latency better.

I'm looking into RMA'ing the drive as quickly as possible. In all my years of purchasing hard drives, this is only the second Western Digital drive I've had to RMA (the first was damaged in shipping). I'm hoping the folks at either Newegg or Western Digital can help with a fast turn around so that we can be back in business.

I'd like to just add that Western Digital makes damn fine products, and it's only because of their products that this work has been possible so far. I personally own seven WD1001FALS (three are used for CHASM), two WD20EADS, two WD6400AAKS, and two WD800 drives. It would not be a stretch to say that I'm a WD fan.

-Robert

When two nodes synchronize via rsync, there is a significant amount of overhead.