Archive Page 2 of 19

« Previous Entries
Next Entries »

One Zope instance and ZEO server with two database

Published by
Jayarajan JN
at April 23, 2008 in Development. 0 

One Zope instance and ZEO server with two database

This setup will introduce the ZEO server using mkzeoinstance and we will have to modify the configuration files, specifically zeo.conf and zope.conf. That’s it! No need to change the code. Connecting and dealing with objects is all the same.

ZEO server is the native solution for database load balancing. it allows us to access the Data.fs over the network. Also, it takes care of object locking!

a. Configuring zeo.conf

In zeo.conf, what is important is to specify the database files it should manage.

    <zeo>
        address 8800
        read-only false
        invalidation-queue-size 100
        # pid-filename $INSTANCE/var/ZEO.pid
        # monitor-address PORT
        # transaction-timeout SECONDS
    </zeo>
    <filestorage 1>
        path $INSTANCE/var/Data1.fs
    </filestorage>
    <filestorage 2>
        path $INSTANCE/var/Data2.fs
    </filestorage>

So now we have our database stored at the ZEO server.

b. Configuring zope.conf

Now we must configure zope.conf to get access to the database via the ZEO server.

    <zodb 1>
        <zeoclient>
        server localhost:8800
            storage 1
            # ZEO client cache, in bytes
            cache-size 20MB
            # Uncomment to have a persistent disk cache
            #client zeo1
        </zeoclient>
    </zodb>
    <zodb 2>
        <zeoclient>
            server localhost:8800
            storage 2
            # ZEO client cache, in bytes
            cache-size 20MB
            # Uncomment to have a persistent disk cache
            #client zeo2
        </zeoclient>
    </zodb>

When the ZEO server and Zope instance are started, the Zope instance will look for the ZEO server and it will connect to the database. The Zope instance will keep on searching at a regular interval if it fails to find the ZEO.

We don’t have to make any changes in the code to migrate from the previous setup to this setup.

One Zope instance with two database

Published by
Jayarajan JN
at April 21, 2008 in Development. 0 

One zope instance two database files

a. Configuring zope.conf

We have to include multiple database files using multiple named <zodb> sections with one <filestorage> key in each which points to the corresponding Data.fs file.

<zodb 1>
    <filestorage>
        path $DATADIR/Data1.fs
    </filestorage>
</zodb>
<zodb 2>
    <filestorage>
        path $DATADIR/Data2.fs
    </filestorage>
</zodb>

We will be referring to the database using the corresponding ZODB label.

b. Connecting to the database programmatically

When Zope starts, it will automatically load the first database entry in zope.conf as default. So now we need to know how we can connect to the second database. The new connection is created using the existing connection.

    connection2 = connection1.get_connection('database_name')

But we need the existing connection to use get_connection(). It is easy if we have an object stored in current database

    siteManager = getSiteManager(some_object)
    defaultFolder = siteManager['default']
    # defaultfolder is an object in first database.
    conn2 = defaultFolder._p_jar.get_connection(’2′)
    conn2.root()['MyFolder'] = Folder()
    transaction.commit()    # myfolder references to an object in the other database
    myfolder=conn2.root()['MyFolder']
    siteManager.registerUtility(myfolder, IFolder)

c. Storing and retrieving the objects

Z3 referencing scheme is very advance to deal with for cross-database referencing. Once we have stored the object, all we need is a reference to access it. Which means, once we have a reference of the object, we don’t need to connect to database before trying to access the object. It will be totally managed by Z3. Retrieval of the object is the same as we do with any other object,

    # Use getUtility to grab the object faster
    folder_in_db2 = getUtility(IFolder)
    folder_in_db2['some_object'] = SomeObject()

You can assure that the object is stored in the second database by verifying the name of database.

    some_object._p_jar.db().database_name

Reference
zope api documentation on cross database references

Multiple database with Zope 3

Published by
Jayarajan JN
at April 21, 2008 in Development. 0 

When we deal with data intensive projects which are expected to be used by a large number of users, we should rethink about the architecture nine times. The best idea is to come up with a flexible, scalable model to build on. Performance, process and database load balancing, and fault-tolerance are the areas that deserve great attention while in the design phase. So how we can achieve such requirements in Zope 3?

It is known that Z3 does not provide any magic features to fully balance the load. However, there is one built-in method that allows us to have multiple identical Zope instances sharing the same database with help of ZEO servers. And for balancing the load among these instances we may use either Pound or Balance. Of course there are other alternatives too!

So what we have on hand to worry about is how to handle the database load. The key requirements that we have to meet are,

  1. High data availability
  2. Fault tolerance
  3. Data safety
  4. Scalability

In Z3, it gives us a few possibilities. And here we will discuss how we can use multiple database with Z3 for different use cases.

  1. One Zope instance with two database
  2. One Zope instance and ZEO server with two database
  3. One Zope instance with multiple ZEO servers
  4. Multiple ZEO servers sharing the same database

These basic setups can be scaled and combined together to design a setup to suit our requirement. All the best!

« Previous Entries
Next Entries »