The Future of Dynamic Cities
with GenAI and Mixed Reality
A collaboration with Generative AI and immersive services from SAP, Communication and
Fraud Prevention APIs from Ericsson/Vonage, and Accenture’s innovation advice.
Timeline
December 2023 –
February 2024
February 2024
Team
Myself, 1 UX Researcher, 1 PM,
2 UX video experts,
client stakeholders
2 UX video experts,
client stakeholders
Role
User experience designer,
workshop facilitator,
storytelling leader
workshop facilitator,
storytelling leader
Contribution
Co-innovation workshop,
findings and storytelling report,
Figma prototype and demo video
findings and storytelling report,
Figma prototype and demo video
Please reach out to me at dylan.rabin@sap.com for more information. View the final output here.
Project Summary
SAP, Accenture, and Ericsson/Vonage presented their partnership at the 2024 Mobile World Congress (MWC), the largest and most influential trade show for the mobile communications industry.
While they had an use case combining product solutions for the field service technician, the scope was too narrow for MWC. They approached our team to reimagine their partnership.
Solution Portfolio
Design Process
Co-innovation Workshop
Following the double diamond of design thinking, we began solving the problem statement by diverging into discovery. This required brainstorming ideas with a variety of different stakeholders from each of the 4 different partnerships and each with their individual goals.
I built and facilitated this workshop with the team by encouraging participants to answer “How Might We” questions we discussed prior with a smaller team of stakeholders. We had about 20 participants across different teams.
Workshop Insights
With plenty of ideas from the workshop participants, the UX researcher and I conslidated all the suggestions into tangible ideas across the different ‘how might we’ questions.
Potential Use Case Directions
After reviewing the brainstorming insights with the UX researcher, we decided to come up with 3 different scenarios that were broader that field service management and applicable to the stakeholders’ needs.
The team reviewed all use cases and decided that the future of city planning would be the most effective because of it would heavily rely on the APIs offered by Ericsson / Vonage and it meets the scalability requirements from SAP / Accenture.
Connected Supply
Chain Optimization
A logistics company notices a delay in a critical shipment. Leveraging advanced technologies, they quickly identify and address the issue. A combination of real-time data, immersive collaboration, and proactive notifications ensures a quick correction and prevention of delays.
Sustainable
Smart Agriculture
An agricultural enterprise aims to utilize modern technology to enhance farming efficiency and sustainability. The goal is to optimize resource use, improve crop yields, and reduce environmental footprint.
Smart City
Resource Efficiency
An urban planning company aims to tackle challenges in efficient resource management and promote sustainable urban development, with a focus on optimizing traffic and reducing energy consumption.
Design + Iterate
With our use case solidified, I began creating designs for the storyline. Since we were limited on time, I was not able to sketch storyboards and instead leveraged current design systems to built the VR experience.
Video Data Stories
Video Data Stories by SAP are bite-sized video-based insights of a business’s data and are shown to viewers similar to an Instagram story. We incorporated a video data story to indicate an alert for the urban planning company.
Immersive Services: Virtual Reality Design
Including virtual reality in this demo offers a powerful tool for businesses to explore, analyze, and communicate insights from SAP’s datasets in an immersive and interactive manner. By leveraging the unique capabilities of VR technology, businesses such as the urban planning company can gain new perspectives on their data and make more informed decisions.
Additionally, opportunities to showcase Vonage APIs with VR were appealing, as Silent Authentication can provide 2FA for the headset, and Quality on Demand and Location APIs can help with connectivity in remote environments without wi-fi.
Generative AI Agent in VR
Including virtual reality in this demo offers a powerful tool for businesses to explore, analyze, and communicate insights from SAP’s datasets in an immersive and interactive manner. By leveraging the unique capabilities of VR technology, businesses such as the urban planning company can gain new perspectives on their data and make more informed decisions.
Additionally, opportunities to showcase Vonage APIs with VR were appealing, as Silent Authentication can provide 2FA for the headset, and Quality on Demand and Location APIs can help with connectivity in remote environments without wi-fi.
Final Video Prototype
After iterating on feedback from stakeholders, we eventually reached our deadline for the Mobile World Congress submissions.
By enhancing the partnership with a broader use case, we successfully catered to a larger target market and were selected to share our work at the conference.
Lessons Learned from Designing in Mixed Reality and for Generative AI
This was one of first projects designing for virtual reality and leveraging generative AI principles in my work. I reflected on my experiences and wrote down key takeaways here.
Virtual Reality Design Takeaways
Spatial Awareness and Scale
- In VR, users interact with digital content in a three-dimensional space, which required looking at spatial awareness and scale carefully.
- For example, design elements needed to be sized and positioned appropriately to provide a sense of depth and immersion without overwhelming or disorienting the user.
Environmental Design
- Unlike traditional 2D interfaces, VR environments can be fully immersive and interactive. We were given the opportunity to create a rich, immersive world that responds to user actions.
- Environmental factors such as lighting, sound, and spatial audio were considered in shaping the user experience and immersion aspects.
Cross-Platform Compatability
- Many traditional 2D UI/UX principles do not directly translate to VR. We needed to rethink interaction patterns and navigation for 3D environments.
- For example, gestural interfaces, spatial menus, and immersive interactions are often preferred over traditional mouse clicks or touchscreen taps.
Generative AI Design Takeaways
Data Integration and Analysis
- Generative AI systems rely on vast datasets to generate content or insights. We needed to ensure seamless integration with SAP BTP and SAP application suite.
- Understanding the specific data requirements and constraints of enterprise was crucial for designing AI systems that can effectively analyze and generate insights from complex datasets.
Interpretability and Explainability
- AI-generated outputs must be interpretable and explainable, particularly in critical decision-making contexts. We incorporated features that provided transparency into Joule’s decision-making process and rationale.
- Visualizations, contextual information, and interactive interfaces can help users understand how Joule’s generated insights are derived and make informed decisions based on them.
Collaboration and Integration
- B2B environments often involve collaboration among multiple stakeholders and systems. We facilitated seamless collaboration and integration by designing AI systems that can interact with existing enterprise tools and workflows.
- APIs, integration frameworks, and interoperable formats can enable AI-generated outputs to be easily integrated into enterprise applications, platforms, or business processes.
