Secure Databricks Deployments for Regulated Industries

Banks and insurers have more data than ever. Yet turning it into value remains hard:

• Core systems, CRM, payments, claims and partner data sit scattered across dozens of disconnected applications.
• Data teams wait for data they can fully trust.
• Reporting, audit and data-residency requirements keep tightening.
• AI-generated fraud and cyberattacks evolve faster than siloed monitoring can follow.

And all of this has to happen under constant cost pressure, with scarce data and AI talent.

The result: underused data, AI pilots that never reach production, and a false choice between compliance gaps and aging legacy systems.

A modern data platform can change that. But in a regulated environment, the question is never only what the platform can do. It is how it is deployed, secured and operated.

Databricks is one of the leading data and AI platforms. It covers the full chain, from data processing and analytics to machine learning and agentic AI (AI systems that can plan and carry out tasks on their own). The platform is evolving quickly, with recent additions such as a managed Postgres database for both analytical and transactional workloads, and an AI Gateway to control and monitor access to AI models.

For regulated organizations, three aspects matter most:

• Enterprise-grade security features that allow secure integration at several layers: network, identity and authorization, and application security.
• An open architecture based on open formats and technologies, which reduces vendor and strategic lock-in.
• One platform for data and AI, meaning one place to integrate data and break silos, and one place to govern access instead of many.

ELCA has deployed Databricks for several financial institutions. For example, Banque Cantonale du Valais (BCVS) started its cloud journey in 2023 with data topics and built a secure data analytics platform on Databricks. This platform has undergone several security audits. We also support a major insurer in Lausanne in modernizing its data integration and AI capabilities, as well as several private banks and asset managers in their data and AI platform initiatives.

We specialize in complex, highly secured deployments, where every data access and every data share must be precisely defined, validated and documented. For organizations that prefer to focus on their data products rather than on operations, our Data Platform as a Service (DPaaS) offers a Databricks platform operated by Everyware (ELCA group), hosted in Switzerland.

Technology is only part of the answer. A successful deployment starts by understanding the available skills, the requirements and the context. It ends with a clear target state: the operating model, the training and upskilling of the teams, and a hypercare period (a phase of reinforced support after go-live) with defined duration and goals.

Compliance frames the whole journey. Upstream, a risk analysis defines the expected security measures, in line with regulatory requirements and data protection laws such as the Swiss FADP (nLPD) and the GDPR. Downstream, audits and penetration tests verify that the measures actually in place match what was promised.

Once the vision is set, a high-level definition of the platform is required. It must answer a few structuring questions:

• What is the platform governance? Who, in business and IT, does what during the project and in operations.
• Where must data reside? Where the data is stored and processed depends on regulatory and internal requirements. This topic is even more complex for AI platforms that rely on hosted LLM.
• How many environments? Typically development, quality assurance and production, each with its own level of protection.
• How many independent workspaces? Separate workspaces can isolate highly confidential data or give business units their own space.
• How does it fit into the existing landscape? The platform must integrate with the network layout and security (firewalls, proxies, data leakage prevention), with the software delivery process, and with application security standards.
• What is the transition plan? How long will the build take? Is it replacing an existing platform? Are there large data volumes to ingest or migrate?

These early decisions shape everything that follows. Changing them later is costly.

A data platform is a complex system with many integrations, and therefore a large attack surface. Its protection comes from several layers working together:

• Network. The platform must fit into the organization’s network security architecture and processes: controlled entry and exit points, audited traffic, and data leakage prevention (tools that detect and block sensitive data leaving the organization).
• Encryption. Data must be protected both when it is sent and when it is stored, to preserve its integrity and confidentiality. Cloud providers always offer a baseline that can be extended with encryption keys controlled by the organization.
• Workspace configuration. Which features are enabled, which compute resources can be used and under which policies, and which roles exist (administrator, contributor, analyst), each with the minimum rights needed.
• Software delivery and application security. Automated deployment pipelines (“CI/CD”) install software, deploy code and assign permissions across the cloud. Because they are so powerful, they must be secured too. This includes the code the organization writes, the third-party packages it relies on, and the roles those pipelines use.
• Catalog architecture. Unity Catalog is the central inventory of data assets and who can use them. It now holds more than tables: AI models, LLM deployments and AI agent skills are also governed there.
• Fine-grained data protection. Row- and column-level security restricts which records and fields each user can see. Tags and attributes classify data (for example “client-identifying”) and trigger access rules based on these attributes.
• Sources and consumers. Every system that feeds data into the platform, and every application or user that reads from it, is an entry point to secure.
• Observability and audit. A secure data platform is proven and maintainable only if logs, monitoring and alerts allow for continuous oversight and audits.

Across all layers, the following principles apply: strong management of identities and authorizations, segregation of duties and least privilege. Each person, application and pipeline gets access only to what it needs, and no one controls an entire chain alone.

Databricks moves fast. More and more capabilities are delivered as serverless or managed services, where the provider runs the infrastructure, for example AI models for inference. This brings agility, but it also extends the architecture and the security perimeter beyond the organization’s own network. Each new service must be assessed, configured and monitored before it is opened to users. Security is therefore not a one-time project, but a continuous practice.

For banks and insurers, a secure data platform is not a brake on innovation. It is what allows AI use cases to leave the pilot stage with confidence. Start with a clear plan, secure every layer, and invest in the people and processes that will run it.