Mird reference manual Contents * Functions + mird_initialize - initialize a database structure + mird_open - open the database + mird_sync - sync the database and journal + mird_sync_please - sync the database and journal + mird_close - close the database + mird_free_structure - frees the database structure + mird_free - frees a database buffer + mird_transaction_new - starts a new transaction + mird_transaction_close - completes the transaction + mird_transaction_cancel - cancels the transaction + mird_tr_resolve - tries to resolve the transaction + mird_tr_free - frees the transaction structure + mird_key_store - store a key:value pair in a table + mird_key_lookup - lookup a key in a table + mird_transaction_key_lookup - lookup a key in a table + mird_key_new_table - create a new table + mird_delete_table - completely delete a table + mird_depend_table - add a table dependency + mird_table_scan - reads entries from a table + mird_transaction_table_scan - reads entries from a table + mird_free_scan_result - frees scan result + mird_s_key_store - store a string key:value pair in a table + mird_s_key_lookup - lookup a string key in a table + mird_transaction_s_key_lookup - lookup a string key in a table + mird_s_key_new_table - create a new string key table + mird_s_table_scan - read entries from a table + mird_transaction_s_table_scan - read entries from a table + mird_free_s_scan_result - frees scan result + mird_free_error - frees MIRD_RES error structure + mird_describe_error - describes a MIRD_RES structure + mird_perror - prints error messages + MIRD_IS_CONFLICT - test if an exception means conflict * Datatypes + struct mird - the database object + struct mird_transaction - the transaction object + mird_key_t and mird_size_t - the database integer types + MIRD_RES - the error or result structure + struct mird_scan_result - result from mird_table_scan + struct mird_s_scan_result - result from mird_s_table_scan * Errors and exceptions * Index Functions mird_initialize - initialize a database structure MIRD_RES mird_initialize(char *base_filename, struct mird **dest); This allocates a new structure for a database object, and initializes it with the default values as well as the filename to use. You can change some values before you call mird_open, see struct mird. mird_open - open the database MIRD_RES mird_open(struct mird *db); This opens the database. The structure to use in this call is created by mird_initialize. Note that a reopening of a database not closed correctly can take some time, since it's cleaned before usage can start. mird_sync - sync the database and journal MIRD_RES mird_sync(struct mird *db); This syncs the database, ie flushes any buffers, syncs against disk, and then starts to reuse any block marked as free. The last operation reads through the journal file for blocks that might be reused, so it can take some time depending on how big your journal file is. The journal file is recreated after this operation. The sync operation fails if there is ongoing transactions, since no block can be reused and therefor the journal file can't be recreated. Call mird_sync_please if you don't know if you have transactions still in progress. mird_sync_please - sync the database and journal MIRD_RES mird_sync_please(struct mird *db); This calls mird_sync immediately, if no transactions in progress. If there is, it sets a flag in the database which will call mird_sync when the last transaction is closed or cancelled. This is useful to call once in a while, for instance once per minute. mird_close - close the database MIRD_RES mird_close(struct mird *db); This syncs the database (see mird_sync) and closes the database, marked as clean. It also frees the database structure. Note that close cancels and unlinks any transaction objects still in progress - the db pointer in the structure sets to NULL - but does not free them. If you might have transactions left after a close, you need to call mird_tr_free to free the transaction structure. mird_free_structure - frees the database structure void mird_free_structure(struct mird *db); This frees the database structure. This function is only to be used only in emergency situations, since the database isn't cleaned before close. mird_free - frees a database buffer void mird_free(unsigned char *buffer); This frees any buffers that is the result of operations like mird_key_lookup and mird_s_key_lookup. mird_transaction_new - starts a new transaction MIRD_RES mird_transaction_new(struct mird *db, struct mird_transaction **mtr); This creates a new transaction, within changes to the database are possible. A transaction can either succeed in total or fail in total. mird_transaction_close - completes the transaction MIRD_RES mird_transaction_close(struct mird_transaction *mtr); This finishes of a transaction. This will return conflict exceptions if this transaction conflicts with any other. It also frees the transaction object (upon success). mird_transaction_cancel - cancels the transaction MIRD_RES mird_transaction_cancel(struct mird_transaction *mtr); This cancels a transaction and frees the transaction object. mird_tr_resolve - tries to resolve the transaction MIRD_RES mird_tr_resolve(struct mird_transaction *mtr); This can be called to synchronize the transaction with the current state of the database. It complains in the same manner as mird_transaction_close, but may be called more then once. mird_tr_free - frees the transaction structure void mird_tr_free(struct mird_transaction *mtr); This frees a transaction object. This is only for those cases when the database is closed but you still have transaction structures. (For instance, when libmird is used in an object oriented environment.) mird_key_store - store a key:value pair in a table MIRD_RES mird_key_store(struct mird_transaction *mtr, mird_key_t table_id, mird_key_t key, unsigned char *value, mird_size_t len); This writes down a key:value pair (where key is a 32 bit integer) in the database, in the given table. If value is NULL, the key:value pair is removed from the table. The table must preexist (mird_key_new_table) and be of the correct type for the operation to succeed. mird_key_lookup - lookup a key in a table MIRD_RES mird_key_lookup(struct mird *db, mird_key_t table, mird_key_t key, unsigned char **data, mird_size_t *len); This looks up a key (integer) in a table. (*data) is set to NULL if the key didn't exist in the table. The resulting data must be freed by a call to mird_free, if it isn't NULL. The table must preexist (mird_key_new_table) and be of the correct type for the operation to succeed. (MIRDE_NO_TABLE and MIRDE_WRONG_TABLE) See also mird_transaction_key_lookup and mird_s_key_lookup. mird_transaction_key_lookup - lookup a key in a table MIRD_RES mird_transaction_key_lookup(struct mird_transaction *mtr, mird_key_t table_id, mird_key_t key, unsigned char **data, mird_size_t *len); This looks up a key (integer) in a table. (*data) is set to NULL if the key didn't exist in the table. The resulting data must be freed by a call to mird_free, if it isn't NULL. The table must preexist (mird_key_new_table) and be of the correct type for the operation to succeed. (MIRDE_NO_TABLE and MIRDE_WRONG_TABLE) The difference against mird_key_lookup is that the lookup operation is transaction-local. This means the state when the transaction is created is kept, and unless mird_tr_resolve is called, no other transaction can change the state of the database from this transaction's point of view. This might be useful to do atomic operations, or to read back data in a transaction not yet closed (or to create a temporary data storage, to later cancel the transaction). mird_key_new_table - create a new table MIRD_RES mird_key_new_table(struct mird_transaction *mtr, mird_key_t table_id); This creates a new table in the database, for use with integer (32-bit) keys; mird_key_lookup and mird_key_store. This fails if the table preexists (MIRDE_TABLE_EXISTS). mird_delete_table - completely delete a table MIRD_RES mird_delete_table(struct mird_transaction *mtr, mird_key_t table_id); This deletes a table and it's contents from the database. The operation fails if the table doesn't exist (MIRDE_NO_TABLE). mird_depend_table - add a table dependency MIRD_RES mird_depend_table(struct mird_transaction *mtr, mird_key_t table_id); This adds a dependency on a table; no other transaction may change this table in any way. This method might be useful if the atomic result is depending on the contents of a whole table, but does not change it. It can be seen as a write-lock on a table. mird_table_scan - reads entries from a table MIRD_RES mird_table_scan(struct mird *db, mird_key_t table_id, mird_size_t n, struct mird_scan_result *prev, struct mird_scan_result **dest); This is used to read all contents from an integer key table, n elements at a time. If prev is NULL, the reading starts from the beginning of the table. Next time around, reuse the old dest as prev to continue reading. (*dest) is set to NULL if there is no more to read. (*dest)->n contains the number of elements actually read, which may differ for string tables. The table must exist and be of the correct type for the operation to succeed. (MIRDE_NO_TABLE and MIRDE_WRONG_TABLE) The prev structure is freed upon call, otherwise it must be freed by calling mird_free_scan_result. Example: struct mird_scan_result *scanner; MIRD_RES res; for (;;) { if ( (res=mird_table_scan(my_database,some_table,100,scanner,&scanner) ) ) [...do something with the error...] if (!scanner) break; /* no more to read */ [...do something with the scan result...] } The key:value pairs read are not in any particular order. mird_transaction_table_scan - reads entries from a table MIRD_RES mird_transaction_table_scan(struct mird_transaction *mtr, mird_key_t table_id, mird_size_t n, struct mird_scan_result *prev, struct mird_scan_result **dest); This is used to read all contents from an integer key table, n elements at a time. If prev is NULL, the reading starts from the beginning of the table. Next time around, reuse the old dest as prev to continue reading. (*dest) is set to NULL if there is no more to read. (*dest)->n contains the number of elements actually read. The table must exist and be of the correct type for the operation to succeed. (MIRDE_NO_TABLE and MIRDE_WRONG_TABLE) The prev structure is freed upon call, otherwise it must be freed by calling mird_free_scan_result. The difference against mird_table_scan is that this reads at the transaction-local point of view, see mird_transaction_key_lookup. mird_free_scan_result - frees scan result void mird_free_scan_result(struct mird_scan_result *msr); This frees the scan structure result from the mird_table_scan functions. See also mird_free_s_scan_result. mird_s_key_store - store a string key:value pair in a table MIRD_RES mird_s_key_store(struct mird_transaction *mtr, mird_key_t table_id, unsigned char *skey, mird_size_t skey_len, unsigned char *value, mird_size_t len); This writes down a key:value pair (where the key is a string of any size) in the database, in the given table. If value is NULL, the key:value pair is removed from the table. The table must preexist (mird_s_key_new_table) and be of the correct type for the operation to succeed. mird_s_key_lookup - lookup a string key in a table MIRD_RES mird_s_key_lookup(struct mird *db, mird_key_t table_id, unsigned char *key, mird_size_t key_len, unsigned char **data, mird_size_t *len); This looks up a key (string) in a table. (*data) is set to NULL if the key didn't exist in the table. The resulting data must be freed by a call to mird_free, if it isn't NULL. The table must preexist (mird_s_key_new_table) and be of the correct type for the operation to succeed. (MIRDE_NO_TABLE and MIRDE_WRONG_TABLE) See also mird_transaction_s_key_lookup and mird_key_lookup. mird_transaction_s_key_lookup - lookup a string key in a table MIRD_RES mird_transaction_s_key_lookup(struct mird_transaction *mtr, mird_key_t table_id, unsigned char *key, mird_size_t key_len, unsigned char **data, mird_size_t *len); This looks up a key (string) in a table. (*data) is set to NULL if the key didn't exist in the table. The resulting data must be freed by a call to mird_free, if it isn't NULL. The table must preexist (mird_s_key_new_table) and be of the correct type for the operation to succeed. (MIRDE_NO_TABLE and MIRDE_WRONG_TABLE) The difference against mird_key_lookup is that the lookup operation is transaction-local. This means the state when the transaction is created is kept, and unless mird_tr_resolve is called, no other transaction can change the state of the database from this transaction's point of view. This might be useful to do atomic operations, or to read back data in a transaction not yet closed (or to create a temporary data storage, to later cancel the transaction). mird_s_key_new_table - create a new string key table MIRD_RES mird_s_key_new_table(struct mird_transaction *mtr, mird_key_t table_id); This creates a new table in the database, for use with string keys; mird_s_key_lookup and mird_s_key_store. This fails if the table preexists (MIRDE_TABLE_EXISTS). mird_s_table_scan - read entries from a table MIRD_RES mird_s_table_scan(struct mird *db, mird_key_t table_id, mird_size_t n, struct mird_s_scan_result *prev, struct mird_s_scan_result **dest); This is used to read all contents from an string key table, approximately n elements at a time. If prev is NULL, the reading starts from the beginning of the table. Next time around, reuse the old dest as prev to continue reading. (*dest) is set to NULL if there is no more to read. (*dest)->n contains the number of elements actually read, which may differ for string tables. The table must exist and be of the correct type for the operation to succeed. (MIRDE_NO_TABLE and MIRDE_WRONG_TABLE) The prev structure is freed upon call, otherwise it must be freed by calling mird_free_s_scan_result. Example: struct mird_scan_result *scanner; MIRD_RES res; for (;;) { if ( (res=mird_s_table_scan(my_database,some_table,100,scanner,&scanne r)) ) [...do something with the error...] if (!scanner) break; /* no more to read */ [...do something with the scan result...] } The key:value pairs read is not in any particular order. mird_transaction_s_table_scan - read entries from a table MIRD_RES mird_transaction_s_table_scan(struct mird_transaction *mtr, mird_key_t table_id, mird_size_t n, struct mird_s_scan_result *prev, struct mird_s_scan_result **dest); This is used to read all contents from an string key table, approximately n elements at a time. If prev is NULL, the reading starts from the beginning of the table. Next time around, reuse the old dest as prev to continue reading. (*dest) is set to NULL if there is no more to read. (*dest)->n contains the number of elements actually read, which may differ. The table must exist and be of the correct type for the operation to succeed. (MIRDE_NO_TABLE and MIRDE_WRONG_TABLE) The prev structure is freed upon call, otherwise it must be freed by calling mird_free_s_scan_result. The difference against mird_s_table_scan is that this reads at the transaction-local point of view, see mird_transaction_s_key_lookup. mird_free_s_scan_result - frees scan result void mird_free_s_scan_result(struct mird_s_scan_result *mssr); This frees the scan structure result from the mird_s_table_scan functions. See also mird_free_scan_result. mird_free_error - frees MIRD_RES error structure void mird_free_error(MIRD_RES error); This frees the error structure contained in MIRD_RES, and must be called if the result from any call is non-NULL. mird_describe_error - describes a MIRD_RES structure void mird_describe_error(MIRD_RES error,char **dest); This creates a human-readable (kinda) message from the error structure. The resulting string may be freed by calling mird_free. mird_perror - prints error messages void mird_perror(char *prefix,MIRD_RES error); This prints out a human-readable message (same as from mird_describe_error) on stderr, preceeded by the given prefix string and a colon (":"). MIRD_IS_CONFLICT - test if an exception means conflict int MIRD_IS_CONFLICT(MIRD_RES res); This macro reports true (1) if the given errors means a conflict, as reported from mird_close. Mird data types: struct mird - the database object This is the main database object. It is created by calling mird_initialize. Some parameters can be modified before calling mird_open, to either set up the behavior or the physical database parameters. * int flags; Possible flags (or them together): + MIRD_READONLY - open database in read-only mode + MIRD_NOCREATE - database file must not be created + MIRD_EXCL - database file must be created (opened with O_EXCL) + MIRD_CALL_SYNC - call sync(2) after fsync + MIRD_SYNC_END - fsync will be called at transaction end + MIRD_JO_COMPLAIN - complain if the database is dirty and the journal file is missing (this is useful if you're for instance is dealing with physically moved databases). + MIRD_NO_KEY_CHECKED_LIST - don't keep a hash table of all keys checked for usage (this is a speedup with about 50%, but takes O[n]+O[m] in memory for n = all keys written since last sync and m is all blocks that might be freed, in opposite of only O[m]) Default is none of these flags. These are the physical database parameters, they may not be changed once the database is created: * int block_size; Database block size; must be even 2^n. Default is 2048. Database size is limited to 2^32 blocks, ie block_size*2^32 (the default limit, 2048*2^(32-5) is approximately 500Gb). * int frag_bits; This sets the number of frags in a fragmented block, and address bits to locate them. Default is 5 bits. The number of frags in a fragmented block is equal to 2^frag_bits-1, the default 2^5-1=32-1=31. This steals bits from the 32 bit address used to point out a chunk. * int hashtrie_bits; This is the number of bits in each hash-trie hash; 4*2^n must be a lot less then block_size. Default is 5, 32 branches at each node. * int file_mode; This is the third argument to open(2), the file mode in which the new database file is created (and journal files). Default is 0666. This is (by the system) masked through umask. And some parameters that exists for optimization and to tune memory usage. Experts only: * int cache_size; This is the number of blocks cached from the database; this is used for both read- and write cache. Note that the memory usage is constant, approximately block_size*cache_size bytes. Default is 128, ie 128 blocks in memory. * int cache_search_length; This is the closed hash maximum search length to find a block in the cache; note that this will result in linear time usage. Default is 8. * int max_free_frag_blocks; this is how many blocks with free space that is kept in a list to be used when a transaction is running. The closest fit of usage of these blocks are where the data will be stored. Default is 10. * int journal_readback_n; This is how many journal entries are read back at once at transaction finish, cancel or copy time bytes needed is approximately 24*this number. Default is 200. There really isn't any other user friendly contents of this structure. See also: mird_initialize, mird_open, mird_close and mird_free_structure. struct mird_transaction - the transaction object This structure doesn't contain any user friendly information, although the element db may be checked to see if the transaction object is connected to a database (ie, use mird_transaction_cancel) or not (ie, use mird_free_structure. In the later case, it is set to NULL. mird_key_t and mird_size_t - the database integer types These two are typedef'ed to be 32 bit unsigned integers, always. They exist as special types and not as for instance "int", since the size of these types may vary between systems and compilers. The 32 bit size is used in the physical database. MIRD_RES - the error or result structure MIRD_RES #defined as pointer to a struct mird_error, which contains information about what has happened. A value of NULL means that everything went as expected. struct mird_error { int error_no; char *s; unsigned x; unsigned y; unsigned z; }; s, x, y and z elements contains different information depending on the given error number (in error_no). (See the error list.) The mird_describe_error and mird_perror exists to get some readable text from this structure. If not NULL, it must always be freed by calling mird_free_error. struct mird_scan_result - result from mird_table_scan struct mird_scan_result { mird_size_t n; - number of actual elements struct mird_scan_tupel { mird_key_t key; - tupel key unsigned char *value; - value data mird_size_t value_len; - value data size } tupel[...]; }; Note that the value data may be unaligned in memory. struct mird_s_scan_result - result from mird_s_table_scan struct mird_s_scan_result { mird_size_t n; - number of actual elements struct mird_s_scan_tupel { unsigned char *key; - key data mird_size_t key_len; - key data size unsigned char *value; - value data mird_size_t value_len; - value data size } tupel[...]; }; Note that both the key or the value data may be unaligned in memory. Mird errors and exceptions: Invalid arguments MIRDE_INVALID_SETUP - invalid database parameters MIRDE_TR_CLOSED - transaction is closed MIRDE_TR_RUNNING - there is ongoing transactions MIRDE_READONLY - database is readonly, s=func MIRDE_NO_TABLE - no such table, x=table id MIRDE_TABLE_EXISTS - table exists already, x=table id MIRDE_WRONG_TABLE - wrong table type, x=table id file open/create/lock general errors s=filename, x=errno MIRDE_OPEN - open failed MIRDE_OPEN_CREATE - open/create failed MIRDE_LOCK_FILE - didn't get lock on file MIRDE_RM - failed to remove file MIRDE_CREATED_FILE - special case returned from mird_open_file, * meaning the file was created (no other info) database file errors x=block no MIRDE_DB_LSEEK - lseek failure, y=errno MIRDE_DB_READ - read failure, y=errno MIRDE_DB_READ_SHORT - read too few bytes, y=bytes, z=exp. MIRDE_DB_WRITE - read failure, y=errno MIRDE_DB_WRITE_SHORT - read too few bytes, y=bytes, z=exp. MIRDE_DB_READ_CHECKSUM - checksum error in block read MIRDE_DB_STAT - call to fstat failed, y=errno MIRDE_DB_SYNC - call to fsync failed, y=errno MIRDE_DB_CLOSE - call to close failed, y=errno Fatal database errors x=block no MIRDE_WRONG_BLOCK - wrong block type; y=exp, z=got MIRDE_ILLEGAL_FRAG - try to read an illegal frag; y=frag MIRDE_ILLEGAL_FREE - link to a non-freelist block MIRDE_WRONG_CHUNK - x=chunk id, wrong type; y=exp, z=got MIRDE_CELL_SHORT - x=chunk id, cell chain broken early MIRDE_SMALL_CHUNK - x=chunk id, y=needed bytes, z=type MIRDE_DATA_MISSING - skey block doesn't contain the data it's supposed to hashtrie errors; x=hashtrie root y=key z=current MIRDE_HASHTRIE_RECURSIVE - too deep recursion (infinite?) MIRDE_HASHTRIE_AWAY - led to key that doesn't match opening an old database while reading header, no other info MIRDE_INVAL_HEADER - given file has an invalid header MIRDE_UNKNOWN_VERSION - can't read that version of database MIRDE_GARBLED - database garbled; need resurrection journal file errors MIRDE_JO_LSEEK - lseek failure, y=errno MIRDE_JO_WRITE - write failure, y=errno MIRDE_JO_WRITE_SHORT - read too few bytes, y=bytes, z=exp. MIRDE_JO_TOO_BIG - journal too big, sync it MIRDE_JO_READ - read failure, x=pos, y=errno MIRDE_JO_MISSING_CANCEL - missing cancel marker just written MIRDE_JO_SYNC - fsync/fdatasync failed, y=errno reading back journal errors MIRDE_MISSING_JOURNAL - dirty and missing journal file MIRDE_VERIFY_FAILED - transaction verification failed resource errors MIRDE_RESOURCE_MEM - out of memory; x=bytes needed MIRDE_RESOURCE_ERROR - failed to allocate error; x=b needed resolving conflicts, x=table_id y=key MIRDE_CONFLICT - conflict MIRDE_DEPEND_BROKEN - broken dependency MIRDE_CONFLICT_S - conflict, s=key, z=len MIRDE_DEPEND_BROKEN_S - broken dependency, s=key, z=len MIRDE_TABLE_DEPEND_BROKEN - broken table dependency, y=table Index * mird_close * mird_delete_table * mird_depend_table * mird_describe_error * mird_free * mird_free_error * mird_free_s_scan_result * mird_free_scan_result * mird_free_structure * mird_initialize * MIRD_IS_CONFLICT * mird_key_lookup * mird_key_new_table * mird_key_store * mird_key_t and mird_size_t * mird_open * mird_perror * MIRD_RES * mird_s_key_lookup * mird_s_key_new_table * mird_s_key_store * mird_s_table_scan * mird_sync * mird_sync_please * mird_table_scan * mird_tr_free * mird_tr_resolve * mird_transaction_cancel * mird_transaction_close * mird_transaction_key_lookup * mird_transaction_new * mird_transaction_s_key_lookup * mird_transaction_s_table_scan * mird_transaction_table_scan * MIRDE_CELL_SHORT * MIRDE_CONFLICT * MIRDE_CONFLICT_S * MIRDE_CREATED_FILE * MIRDE_DATA_MISSING * MIRDE_DB_CLOSE * MIRDE_DB_LSEEK * MIRDE_DB_READ * MIRDE_DB_READ_CHECKSUM * MIRDE_DB_READ_SHORT * MIRDE_DB_STAT * MIRDE_DB_SYNC * MIRDE_DB_WRITE * MIRDE_DB_WRITE_SHORT * MIRDE_DEPEND_BROKEN * MIRDE_DEPEND_BROKEN_S * MIRDE_GARBLED * MIRDE_HASHTRIE_AWAY * MIRDE_HASHTRIE_RECURSIVE * MIRDE_ILLEGAL_FRAG * MIRDE_ILLEGAL_FREE * MIRDE_INVAL_HEADER * MIRDE_INVALID_SETUP * MIRDE_JO_LSEEK * MIRDE_JO_MISSING_CANCEL * MIRDE_JO_READ * MIRDE_JO_SYNC * MIRDE_JO_TOO_BIG * MIRDE_JO_WRITE * MIRDE_JO_WRITE_SHORT * MIRDE_LOCK_FILE * MIRDE_MISSING_JOURNAL * MIRDE_NO_TABLE * MIRDE_OPEN * MIRDE_OPEN_CREATE * MIRDE_READONLY * MIRDE_RESOURCE_ERROR * MIRDE_RESOURCE_MEM * MIRDE_RM * MIRDE_SMALL_CHUNK * MIRDE_TABLE_DEPEND_BROKEN * MIRDE_TABLE_EXISTS * MIRDE_TR_CLOSED * MIRDE_TR_RUNNING * MIRDE_UNKNOWN_VERSION * MIRDE_VERIFY_FAILED * MIRDE_WRONG_BLOCK * MIRDE_WRONG_CHUNK * MIRDE_WRONG_TABLE * struct mird * struct mird_s_scan_result * struct mird_scan_result * struct mird_transaction