259 results about "Distributed data store" patented technology

Various method and system embodiments of the present invention are directed to hierarchical control logic within each component data-storage system of a distributed data-storage system composed of networked component data-storage systems over which virtual disks, optionally replicated as virtual-disk images, composed of data segments in turn composed of data blocks, are distributed at the granularity of segments. Each data segment is distributed according to a configuration. The hierarchical control logic includes, in one embodiment of the present invention, a top-level coordinator, a virtual-disk-image-level coordinator, a segment-configuration-node-level coordinator, a configuration-group-level coordinator, and a configuration-level coordinator.

Embodiments of the present invention are directed to methods for maintaining data consistency of data blocks during migration or reconfiguration of a current configuration within a distributed data-storage system to a new configuration. In one embodiment of the present invention, the current configuration is first determined to be reconfigured. The new configuration is then initialized, and data blocks are copied from the current configuration to the new configuration. Then, the configuration states maintained by component data-storage systems that store data blocks of the current and new configurations are synchronized. Finally, the current configuration is deallocated. In a second embodiment of the present invention, a current configuration is determined to be reconfigured, and, while carrying out continuing READ and WRITE operations directed to data blocks of the current configuration in a data-consistent manner, the new configuration is initialized, data blocks are copied from the current configuration to the new configuration, and the timestamp and data states for the data blocks of the current and new configurations are synchronized.

A method for storing and retrieving data in a storage node of a data store and storage node of a data store, storing in main-memory at least one segment of a relational table The storage node includes at least one computational core running at least one scan thread each dedicated to the scanning of one of the at least one segment. At least one scan thread uniquely, continuously and exhaustively scans the dedicated segment of the relational table. The storage node receives and processes batches of query and update operations for the at least one segment of the relational table. The query and update operations of a batch are re-indexed at beginning of each scan by the scan thread. Then, the indexed query and update operations of a batch are independently joined to data records of said segment that match with predicates of the indexed query and update operations.

The present invention generally relates to distributed data storage systems. In particular, the present invention is related to a method of data storing in a distributed data storage system that comprises a clustering of data blocks and the use of random linear combinations of data blocks that makes the distributed data store system efficient in terms of storage space needed and inter-device communication costs, both for the storage method, as for the associated repair method.

An archival storage cluster of preferably symmetric nodes includes a metadata management system that organizes and provides access to given metadata, preferably in the form of metadata objects. Each metadata object may have a unique name, and metadata objects are organized into regions. Preferably, a region is selected by hashing one or more object attributes (e.g., the object's name) and extracting a given number of bits of the resulting hash value. The number of bits may be controlled by a configuration parameter. Each region is stored redundantly. A region comprises a set of region copies. In particular, there is one authoritative copy of the region, and zero or more backup copies. The number of backup copies may be controlled by a configuration parameter. Region copies are distributed across the nodes of the cluster so as to balance the number of authoritative region copies per node, as well as the number of total region copies per node. Backup region copies are maintained synchronized to their associated authoritative region copy.

Data management systems and methods for distributed content storage and management using content signatures that use file identicality properties are provided. A data management system is provided that includes a content engine for managing the storage of file content, a content signature generator that generates a unique content signature for a file processed by the content engine, a content signature comparator that compares content signatures and a content signature repository that stores content signatures. Methods are provided for the efficient management of files using content signatures that take advantage of file identicality properties. Content signature application modules and registries exist within information source clients and centralized servers to support the content signature methods.

Various embodiments of the present invention provide methods, in distributed data-storage systems that associate one or more timestamps with each data block in each data-storage-component, for deciding whether or not a data block has been written. In certain embodiments of the present invention, a sparse database of timestamps associated with data blocks is maintained, each timestamp having a field that contains one of an indication of a time or sequence and a sentinel value indicating that the timestamp is garbage collected. When a timestamp is not found associated with a data block in a timestamp database, the data block is associated with a garbage-collected-timestamp state. In various embodiments of the present invention, data structures are maintained that store status information indicating whether or not any of a number of data blocks in each of a number of data-block-allocation units have been written. During replication, migration, or reconfiguration of a current segment of data blocks to a new segment of data blocks in these various embodiments, a data block is determined to be written or to be unwritten by determining, from the data structures, whether or not a data block-allocation unit containing the data block is written or unwritten.

A distributed data storage system and method comprising a highly integrated mass storage controller system permitting distributed management and control of data storage is disclosed. The present invention in some preferred embodiments permits mass storage media to be made available on a network in a fashion to permit global access, while automatically handling many high-level file access and data integrity / security / control functions normally associated with a host operating system. This integration and redistribution of functionality permits spatial diversification of data resources, automatic mirroring of data, fault isolation, data security, and a plethora of other control functions to be integrated into the mass storage device. This integration permits peer-to-peer communication between mass storage devices to both unburden the host data consumers but also isolate the data management functions from the data presentation functions normally associated with host systems. Exemplary embodiments of the present invention as applied to specific preferred system contexts include but are not limited to distributed data storage in a networked environment, brokered data access metering, database access / control, data backup, journaling, checkpointing, and automated software distribution.

Various embodiments relate generally to data science and data analysis, computer software and systems, and wired and wireless network communications to provide an interface between repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform that facilitates consolidation of one or more datasets, whereby a collaborative data layer and associated logic facilitate, for example, efficient access to, and implementation of, collaborative datasets. In some examples, a method may include receiving data representing a query of a consolidated dataset that may include datasets formatted atomized datasets, analyzing the query to classify portions of the query to form classified query portions, partitioning the query into sub-queries as a function of a classification type for each of the classified query portions, and retrieving data representing a query result from distributed data repositories.

The disclosed embodiments disclose techniques for synchronizing file updates between two cloud controllers of a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in the cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller receives a request from a client to access a file in the distributed filesystem. The cloud controller sends a synchronization update request for the file to a second cloud controller and in response receives a synchronization update for the file from the second cloud controller.

A system and method for versioning of atomic units of data (such as XML fragments) in a distributed data store are disclosed. The distributed data store may maintain a linear version history for each fragment, such that each has no more than one successor and one predecessor, with no implicit branching. An application programming interface may provide an explicit branching request (indicating an intent to merge modified data in a child branch into its parent branch), along with read, create, create-merge, and / or integrate requests. A reconciliation policy may be specified on a per-application basis, and may include automatic reconciliation using a callback function or interactive reconciliation, with inputs returned to the application. Writes to a minority side of a network partition may be refused unless an explicit branch is requested. Transactions may not be supported in client applications or may be supported through explicit branching. Transactions may be used internally.

Version indicators within an existing range can be associated with a data partition in a distributed data store. A partition reconfiguration can be associated with one of multiple partitions in the data store, and a new version indicator that is outside the existing range can be assigned to the reconfigured partition. Additionally, a broadcast message can be sent to multiple nodes, which can include storage nodes and / or client nodes that are configured to communicate with storage nodes to access data in a distributed data store. The broadcast message can include updated location information for data in the data store. In addition, a response message can be sent to a requesting node of the multiple nodes in response to receiving from that node a message that requests updated location information for the data. The response message can include the requested updated location information.

Provided is a process including: receiving a request to access a distributed data store, wherein the distributed data store stores a plurality of units of content that are each distributed among multiple computing entities hosting different subsets of data of the distributed data store; and causing, with one or more processors of a computing device configured to participate in combining the information from the plurality of computing entities to access units of content, logging of the request in an entry in a tamper-evident log.

Various embodiments of the present invention are directed to distributed data-storage systems that provide non-distributed-data-storage-device-like interfaces to host computers that access the distributed data-storage systems. Distributed data-storage systems generally rely on state information used for execution of commands, received from host computers, by component data-storage systems. Command-executing component data-storage systems may access and modify state information shared among many or all of the component data-storage systems. Embodiment methods of the present invention provide methods for partitioning state information within a distributed data-storage system as well as communications protocols and methods for managing shared state information among the component data-storage systems within a distributed data-storage system. In certain embodiments of the present invention, state information is partitioned into one or more of: (1) local state information that is managed, accessed, and modified separately by each component data storage system; (2) shared state information that is locally cached on component data storage systems for immediate read access, that is periodically refreshed, but that is kept globally consistent among component data storage systems by distributing state-modifying operations; and (3) continuously consistent, shared state information.

A file protection scheme for fixed content in a distributed data archive uses computations that leverage permutation operators of a cyclic code. In an illustrative embodiment, an N+K coding technique is described for use to protect data that is being distributed in a redundant array of independent nodes (RAIN). The data itself may be of any type, and it may also include system metadata. According to the invention, the data to be distributed is encoded by a dispersal operation that uses a group of permutation ring operators. In a preferred embodiment, the dispersal operation is carried out using a matrix of the form [IN<sub2>—< / sub2>C] where IN is an n×n identity sub-matrix and C is a k×n sub-matrix of code blocks. The identity sub-matrix is used to preserve the data blocks intact. The sub-matrix C preferably comprises a set of permutation ring operators that are used to generate the code blocks. The operators are preferably superpositions that are selected from a group ring of a permutation group with base ring Z2.

A hybrid quorum / consensus and primary-backup fault-tolerance model in an object-based distributed data storage system. When a primary manager fails, a hierarchy of network entities is established in which a group of realm managers first authorizes a failure-handling event through quorum / consensus and a backup manager (for the failing primary manager) then executes the decision of the quorum of realm managers. The realm managers, operating by consensus, determine whether (a) the primary manager can indeed be asserted to be down, and (b) whether there is a quorum of realm managers in agreement on this decision. If both are true, a master realm manager instructs the backup manager to proceed to execute the necessary steps to become the primary manager and function as the primary until the original primary manager is brought back into service. The hybrid fault-tolerance approach handles both single unit failures and network partitions in a unified way, without creating a single cluster out of the fault domain.

Various systems, methods and apparatuses for intelligently and dynamically allocating proxy backup clients to back up and restore vast amounts of data distributed across a network are presented. One method involves identify distributed data stored in a computer system, requesting the data, and performing a backup or restore operation on the data. Other aspects include intelligently dynamically allocating proxy backup clients to perform backup and restore operations on data distributed across a network, and intelligently and dynamically balancing, or re-balancing or otherwise managing the workload among the proxy backup clients.

Load balancing includes receiving, from a client, a connection request to establish a connection with a server; determining load balancing state information based at least in part on the connection request; synchronizing the determined load balancing state information across a plurality of service engines, including to invoke an atomic read-miss-create (RMC) function on a distributed data store service; and distributing the connection to a selected server among a plurality of servers according to a result of the RMC function.

Embodiments include a database controller for a database of information encoded as a set of data items, the database controller comprising: a segment control module configured to divide the set of data items into a series of segments, each segment comprising each of the data items falling within defined lower limit and upper limit values of an ordering metric used to place the data items in order; and an adaptive segment locator module configured to manage the distribution of the series of segments among a plurality of storage units, including selecting which segments from among the series of segments to co-locate on the same storage unit based on historical data representing read operations performed on the database.

Described herein are methods and systems for locating and aggregating data from disparate sources. Data related to an audience of media consumers is accessed from a plurality of sources. At least a portion of the data from the plurality of sources is aggregated into a unified data store, data from the unified data store is extracted at least partly based on significance information.

The present invention relates to a distributed data storage system comprising a plurality of storage nodes. Using unicast and multicast transmission, a server application may write data in the storage system. When writing data, at least two storage nodes are selected based in part on a randomized function, which ensures that data is sufficiently spread to provide efficient and reliable replication of data in case a storage node malfunctions.

The embodiment of the invention provides a distributed file system, and a data storage processing method and a data storage processing device thereof. The method comprises the following steps of: receiving an access request sent by a client of the distributed file system; determining a storage unit to be accessed by the client of the distributed file system and the status of a virtual resource pool corresponding to the storage unit according to the access request, wherein same data is stored in the storage unit and the virtual resource pool corresponding to the storage unit and the virtual resource pool is generated by clustering according to the client of the distributed file system having a same activity cycle; and returning an access response message including a client identifier to the client of the distributed file system initiating the access request, when determining that the virtual resource pool corresponding to the storage unit is in an on-line status, wherein the client of the distributed file system identified by the client identifier is one of the virtual resource pools. The invention further provides the corresponding device and the distributed file system. By the technical scheme, the system stability of the distributed file system can be improved.

A distributed data system may include nodes each including a process that may access distributed data. A lock mechanism coupled to the distributed data store may grant locks for portions of the distributed data. Only threads of a process holding a lock for a portion of the distributed data may access the portion. A process may obtain a lock for a distributed data portion. In one embodiment, the process may release the lock if no more threads require access to the portion so may initiate a "dummy" thread requester indicating a thread requiring access to the portion. If other threads requiring access are initiated, it may not be necessary to obtain the lock again. This may help reduce the use of resources associated with using locks. If the process receives a request to release the lock, the process may cancel the dummy thread requester and release the lock.

A distributed data store may maintain versioned hierarchical data structures. Different versions of a hierarchical data structure may be maintained consistent with a transaction log for the hierarchical data structure. When access requests directed to the hierarchical data structure are received, a version of the hierarchical data structure may be identified for processing an access request. For access requests with snapshot isolation, the identified version alone may be sufficient to consistently process the access request. For access requests with higher isolation requirements, such as serializable isolation, transactions based on the access request may be submitted to the transaction log so that access requests resulting in committed transactions may be allowed, whereas access requests resulting in conflicting transactions may be denied.

The data vaporizer provides secure online distributed data storage services that securely store and retrieve data in a public distributed storage substrate such as public cloud. The data vaporizer vaporizes (e.g., fragmented into tiny chunks of configurable sizes) data and distributes the fragments to multiple storage nodes so that the data is not vulnerable to local disk failures, secures data so that even if some of the storage nodes are compromised, the data is undecipherable to the attacker, stores data across multiple cloud storage providers and / or parties using keys (e.g., tokens) provided by multiple parties (including the owners of the data) and maintains data confidentiality and integrity even where one or more data storage provider is compromised. The data vaporizer is configurable for different domain requirements including data privacy and anonymization requirements, encryption mechanisms, regulatory compliance of storage locations, and backup and recovery constraints.

The invention provides a forest resource heterogeneous data distributed management system. The system comprises a data management module, a service management module, a data service release module, a metadata service module, a metadata registration module and a catalogue and index module, wherein the data management module maintains distributed data stored locally by a data provider and collected by a network; the service management module provides service data; the data service release module acquires and matches the distribution data and the service data so as to generate data service release information, and releases the dynamic deployment of the data service release information to a server; the metadata service module generates data service metadata according to the data service release information; the metadata registration module eliminates the isomerism of the data service metadata through the conversion of a private format, and generates standard metadata of the system; and the catalogue and index module provides catalogues and indexes for the standard metadata of the system. The forest resource heterogeneous data distributed management system realizes the distributed management of the forest resource heterogeneous data of a forestry center.

Data storage reliability is maintained in a write-back distributed data storage system including multiple nodes. Information is stored as a stripe including a collection of a data strips and associated parity strips, the stripe distributed across data and parity nodes. Each data node maintains the data strip holding a first copy of data, and each parity node maintains a parity strip holding a parity for the collection of data strips. A driver node initiates a full-stripe-write parity update protocol for maintaining parity coherency in conjunction with other nodes, to keep the relevant parity strips coherent. Parity is determined directly by computing parity strips for all data strips of a stripe. Any node may function as a driver node.

A distributed data store labels operations with globally unique identifiers that contain approximate timestamps. The labels are used to address causes of inconsistency in the distributed data store while ensuring progress. A first mode is provided that stores the latest label for each entry is useful if re-inserts and deletes are rare. Another mode is provided that stores a history of labels for each entry can be used if there are many re-inserts and deletes. A further mode is provided that stores a history of labels for queries can report updates to query answers as inserts and deletes settle across the distributed data store.

A data storage and retrieval method and system. The method includes defining, by a computing system, a meta-object, a schema based structured definition for the meta-object, and a taxonomy with configuration data. The computing system creates content associated with the taxonomy and associates topics of the content with a logical storage room representations. Reference coordinates associated with the logical storage room representations are associated with the taxonomy. The computing system generates updated configuration data comprising the reference coordinates and a uniform resource identifier associated with the content is generated. The computing system generates an account associated with the meta-object for a user. Metering charges for usage of the account and a report associated with the account and usage are generated.