AWS S3 Vector Search End-to-End Guide
Why Vector Databases Were Created The rise of AI and machine learning has fundamentally changed how we process and understand data. Traditional databases excel at storing structured data like names, numbers, and categories, but they struggle with the complexity of modern AI applications that need to understand: The AI Data Challenge Modern AI systems convert unstructured data (text, images, audio) into vector embeddings – numerical representations that capture semantic meaning. For example: Mathematical Foundation Vector databases solve the Approximate Nearest Neighbor (ANN) problem: What is a Vector Database? A vector database is a specialized database designed to store, index, and query high-dimensional vector embeddings efficiently. Unlike traditional databases that store discrete values, vector databases store continuous numerical representations of data that capture semantic meaning and relationships. Core Components Traditional Database vs Vector Database Example: Understanding Vector Similarity When to Use Vector Databases Perfect for: Not ideal for: Overview Amazon S3 Vectors integrates with OpenSearch to provide flexible vector storage and search capabilities, offering up to 90% cost reduction compared to conventional approaches while seamlessly integrating with Amazon Bedrock Knowledge Bases, SageMaker, and OpenSearch for AI applications. What’s New and Important Amazon S3 Vectors is a new cloud object store that provides native support for storing and querying vectors at massive scale, offering up to 90% cost reduction compared to conventional approaches while seamlessly integrating with Amazon Bedrock Knowledge Bases, SageMaker, and OpenSearch for AI applications. Amazon OpenSearch Serverless now supports half (0.5) OpenSearch Compute Units (OCUs) for indexing and search workloads, cutting the entry cost in half. Architecture Options Option 1: S3 Vectors + OpenSearch Serverless (Recommended) Option 2: OpenSearch Serverless Only Prerequisites: Part 1: Setting Up S3 Vectors 1.1 Create S3 Vectors Bucket 1.2 Upload Vector Data # upload_vectors.pyimport boto3import numpy as npimport json def upload_vectors_to_s3(): s3_client = boto3.client(‘s3’) bucket_name = ‘my-vector-bucket-2024’ # Sample vector data (768-dimensional vectors) vectors = [ { “id”: “doc_1”, “vector”: np.random.rand(768).tolist(), “metadata”: { “title”: “Document 1”, “category”: “technology”, “content”: “AI and machine learning concepts” } }, { “id”: “doc_2”, “vector”: np.random.rand(768).tolist(), “metadata”: { “title”: “Document 2”, “category”: “science”, “content”: “Physics and quantum mechanics” } } ] # Upload vectors for i, vector_data in enumerate(vectors): key = f”vectors/{vector_data[‘id’]}.json” s3_client.put_object( Bucket=bucket_name, Key=key, Body=json.dumps(vector_data), ContentType=’application/json’ ) print(f”Uploaded vector {vector_data[‘id’]}”) if __name__ == “__main__”: upload_vectors_to_s3() Part 2: Setting Up OpenSearch Serverless 2.1 Create OpenSearch Serverless Collection # Create security policyaws opensearchserverless create-security-policy \ –name vector-search-policy \ –type encryption \ –policy ‘{ “Rules”: [ { “ResourceType”: “collection”, “Resource”: [“collection/vector-search-*”] } ], “AWSOwnedKey”: true }’ # Create network policyaws opensearchserverless create-security-policy \ –name vector-search-network-policy \ –type network \ –policy ‘[ { “Rules”: [ { “ResourceType”: “collection”, “Resource”: [“collection/vector-search-*”] }, { “ResourceType”: “dashboard”, “Resource”: [“collection/vector-search-*”] } ], “AllowFromPublic”: true } ]’ # Create access policyaws opensearchserverless create-access-policy \ –name vector-search-access-policy \ –type data \ –policy ‘[ { “Rules”: [ { “ResourceType”: “collection”, “Resource”: [“collection/vector-search-*”], “Permission”: [“aoss:*”] }, { “ResourceType”: “index”, “Resource”: [“index/vector-search-*/*”], “Permission”: [“aoss:*”] } ], “Principal”: [“arn:aws:iam::YOUR_ACCOUNT_ID:root”] } ]’ # Create the collectionaws opensearchserverless create-collection \ –name vector-search-collection \ –type VECTORSEARCH \ –description “Vector search collection for embeddings” 2.2 Create Vector Index # create_index.pyfrom opensearchpy import OpenSearch, RequestsHttpConnectionfrom aws_requests_auth.aws_auth import AWSRequestsAuthimport boto3def create_vector_index(): # Get credentials credentials = boto3.Session().get_credentials() awsauth = AWSRequestsAuth(credentials, ‘us-east-1’, ‘aoss’) # OpenSearch client client = OpenSearch( hosts=[{‘host’: ‘your-collection-endpoint.us-east-1.aoss.amazonaws.com’, ‘port’: 443}], http_auth=awsauth, use_ssl=True, verify_certs=True, connection_class=RequestsHttpConnection ) # Create index with vector mapping index_name = “vector-search-index” index_body = { “settings”: { “index”: { “knn”: True, “knn.algo_param.ef_search”: 100
