Add exporting individual attributes

Based off #32

Also:
* rework README.md
* improve configuration error messages

README.md

@@ -1,16 +1,38 @@
-# Attribute Endpoints Provider
+# Attribute Endpoints Provider - Export User Attributes
 
-This is a Keycloak Provider that exports an anonymized list of user profile attribute values.
+This is a Keycloak Provider that exports user profile attribute values.
-For this it will provide API endpoints for every configured attribute-group.
+The selection of attributes and authorization is manage through an endpoint configuration.
-The configuration of the provider is possible via an admin page.
 
-Every endpoint responds with a list of all attribute values, that:
+## Usage
-- are in the attribute group matching `attribute-group`
+
-- match an optional RegEx Pattern `attribute-regex`
+The provider adds an object type "Attribute Endpoints" to the Keycloak admin website.
-- belong to a user with a role matching `match-role`
+You configure the provider by creating one or more endpoint configurations.
-- are non-empty
+
+Each endpoint configuration requires:
+- a name that is the reference in the endpoint URL (slug)
+- the name of a role that the calling user must have to be allowed to query this endpoint
+- the name of an attribute or an attribute group that should be exported
+- the name of a role that users must have to be included in the list
+- an optional regular expression that must match to have a value included
+
+> **Note** The attribute (group) and the roles need to exist before you can create the endpoint configuration.
+
+There are two endpoints returning JSON:
+- `export/slug`: collate all attribute values into a single list
+- `export/slug/map`: produce a map of lists, with the key being the attribute name
+
+No other data is included in the response, in particular, there is no information on which attribute value is associated with which user.
+
+User Attribute values can be single or multi-value; the resulting list will include them as a flattened list.
+
+> **⚠️ Note** The authorization and the selection of the user attributes are not tied together in any way.
+> Every user that has the matching role will have all the attributes included that are specified in the endpoint configuration.
+> In other words, if you are exporting an attribute group, all attributes will be included.
+
+> **⚠️ Note** Keycloak has no concept of authorization for individual user attributes (for example, based on assigned roles).
+> Users will be able to see and edit any attribute that has been made available to users, irrespective of whether a user would
+> be included in an endpoint configuration export or not.
 
-Multivalue attributes are flattened in the response.
 
 ## Building
 
@@ -20,79 +42,8 @@ Once all dependencies are met, simply call `make` to build the provider, which s
 
 There's also `make clean` available for removing the output directory.
 
+To add the provider to your Keycloak install, copy `attribute-endpoints-provider/target/attribute-endpoints-provider-1.0-SNAPSHOT.jar` to the Keycloak Provider directory (`/opt/keycloak/providers/`).
+
 ## Testing Setup
 
 See [testing/README.md](testing/README.md) for details on the Docker Compose based setup that includes a realm and the ability to attach a debugger to the running Keycloak.
-
-## Example Setup
-
-We assume an unconfigured, fresh Keycloak installation running under `http://localhost:8080`.  
-(This can be achieved by running the provided `compose.yaml` after building the provider as outlined in [Building](#building).)
-
- 1. Add a new realm  
-    e.g. "TestRealm"
- 2. Under `Realm Settings > User profile > Attributes Group`, add a new attribute Group  
-    Example:
-    - `Name` = `"my-attributes-group"`
-    - `Display name` = `"Endpoint Attributes"`
-    - `Display description` = `"Attributes exported by the provider."`
- 3. Under `Realm Settings > User profile > Attributes`, add a new attribute  
-    Example:
-    - `Attribute [Name]` = `"ssh-keys"`
-    - `Display name ` = `"SSH Keys"`
-    - `Multivalued` = `On`
-    - `Attribute group` = `"my-attributes-group"`
-    - `Who can edit?` = `user, admin`
-    - `Validators`  
-      You can add validators, which will limit what values the user can enter. These validators are ignored by the provider.
- 4. Under `Realm roles`, add two new roles  
-    Example:
-    1. `Role name` = `"myattribute-match"`
-    2. `Role name` = `"myattribute-export"`
- 5. Under `Users`, add a new user  
-    Example:
-    - `Username` = `"user"`
-    - `Email` = `"user@example.com"`
-    - `First name` = `"User"`
-    - `Last name` = `"User"`
-    - `SSH Keys` = `"example-value-1", "example-value-2"`
- 6. In the Settings of the newly created user, go to `Role mapping > Assing role > Realm roles` and check the role `myattribute-match`
- 7. create a second user to use the provider
-    - `Username` = `"bot-user"`
-    - `Email` = `"bot@example.com"`
-    - `First name` = `"Bot"`
-    - `Last name` = `"Bot"`
-    - After creating:
-      - give it the role `myattribute-export`
-      - set a password in the users settings `Creadentials > Set password`. For Example `"password"`
-8. Under `Attribute Endpoints > Create item`, add a new endpoint to the provider  
-   Example:
-   - `Slug` = `"ssh_keys"`
-   - `Attribute Group` = `"my-attributes-group"`
-   - `Match Role` = `"myattribute-match"`
-   - `Auth Role` = `"myattribute-export"`
-   - `Attribute RegEx` = `".*"`
-9. Aquire an OIDC Access Token:  
-    ```shell
-    curl --request POST \
-        --url http://localhost:8080/realms/TestRealm/protocol/openid-connect/token \
-        --header 'content-type: application/x-www-form-urlencoded' \
-        --data scope=openid \
-        --data username=bot-user \
-        --data password=password \
-        --data grant_type=password \
-        --data client_id=admin-cli
-    ```
-10. copy the value of the response key `access_token` and use it in a second request:  
-    ```shell
-    curl --request GET \
-        --url http://localhost:8080/realms/TestRealm/attribute-endpoints-provider/export/ssh_keys \
-        --header 'authorization: Bearer ey...' \
-        --header 'content-type: application/json'
-    ```
-11. You should get a response like this:
-    ```json
-    ["example-value-1","example-value-2"]
-    ```
-
-Although this example uses a simple bot account to authenticate to Keycloak, we recommend using a client with service account, when using this provider programmatically.

attribute-endpoints-provider/src/main/java/de/ccc/hamburg/keycloak/attribute_endpoints/AdminUiPage.java

@@ -1,15 +1,12 @@
 package de.ccc.hamburg.keycloak.attribute_endpoints;
 
-import java.util.List;
+import com.google.auto.service.AutoService;
-import java.util.regex.Pattern;
-
 import org.keycloak.Config;
 import org.keycloak.component.ComponentModel;
 import org.keycloak.component.ComponentValidationException;
 import org.keycloak.models.KeycloakSession;
 import org.keycloak.models.KeycloakSessionFactory;
 import org.keycloak.models.RealmModel;
-import org.keycloak.models.RoleModel;
 import org.keycloak.provider.ProviderConfigProperty;
 import org.keycloak.provider.ProviderConfigurationBuilder;
 import org.keycloak.representations.userprofile.config.UPConfig;
@@ -17,7 +14,8 @@ import org.keycloak.services.ui.extend.UiPageProvider;
 import org.keycloak.services.ui.extend.UiPageProviderFactory;
 import org.keycloak.userprofile.UserProfileProvider;
 
-import com.google.auto.service.AutoService;
+import java.util.List;
+import java.util.regex.Pattern;
 
 /**
  * Implements UiPageProvider to show a config page in the admin
@@ -43,6 +41,7 @@ public class AdminUiPage implements UiPageProvider, UiPageProviderFactory<Compon
         return PROVIDER_ID;
     }
 
+    @Override
     public String getHelpText() {
         return "Configure endpoints of the Attribute Endpoint Provider.";
     }
@@ -50,7 +49,7 @@ public class AdminUiPage implements UiPageProvider, UiPageProviderFactory<Compon
     @Override
     public void validateConfiguration(KeycloakSession session, RealmModel realm, ComponentModel model) {
         String errorString = "\n";
-        Boolean hasError = false;
+        boolean hasError = false;
 
         Pattern slugPattern = Pattern.compile("^[a-zA-Z0-9_-]*$");
         String configAttributeSlug = model.getConfig().getFirst("slug");
@@ -86,13 +85,15 @@ public class AdminUiPage implements UiPageProvider, UiPageProviderFactory<Compon
         if (configAttributeGroup == null) {
             hasError = true;
             errorString += " • [Attribute Group] can not be empty\n";
-        } else if (!upconfig.getGroups().stream().anyMatch(g -> g.getName().equals(configAttributeGroup))) {
+        } else if (upconfig.getAttributes().stream().noneMatch(a ->
+                a.getName().equals(configAttributeGroup)
+                        || (a.getGroup() != null && a.getGroup().equals(configAttributeGroup)))) {
             hasError = true;
-            errorString += " • [Attribute Group] does not exist\n";
+            errorString += " • [Attribute Group] no matching Attribute or Attribute Group\n";
         }
 
         String configAttributeRegex = model.getConfig().getFirst("attribute-regex");
-        Boolean regexIsBlank = configAttributeRegex == null;
+        boolean regexIsBlank = configAttributeRegex == null;
 
         if (!regexIsBlank) {
             try {
@@ -113,37 +114,37 @@ public class AdminUiPage implements UiPageProvider, UiPageProviderFactory<Compon
         return ProviderConfigurationBuilder.create()
                 .property()
                 .name("slug")
-                .label("Slug")
+                .label("Endpoint URL Slug")
                 .helpText(
                         "The slug in the path of the API endpoint (e.g. /realms/:realm/attribute-endpoint-provider/export/:slug)")
                 .type(ProviderConfigProperty.STRING_TYPE)
                 .add()
 
+                .property()
+                .name("auth-role")
+                .label("Endpoint Role")
+                .helpText("Calling this endpoint configuration requires an authenticated user that has this role.")
+                .type(ProviderConfigProperty.STRING_TYPE)
+                .add()
+
                 .property()
                 .name("attribute-group")
-                .label("Attribute Group")
+                .label("Attribute Name or Group")
-                .helpText("The attribute group to export.")
+                .helpText("The attribute or attribute group to export.")
                 .type(ProviderConfigProperty.STRING_TYPE)
                 .add()
 
                 .property()
                 .name("match-role")
-                .label("Match Role")
+                .label("User Match Role")
-                .helpText("Export only attributes of users with this role.")
+                .helpText("Only users with this role will have the attributes exported.")
-                .type(ProviderConfigProperty.STRING_TYPE)
-                .add()
-
-                .property()
-                .name("auth-role")
-                .label("Auth Role")
-                .helpText("Role needeed by the authenticated account to be able to use this endpoint.")
                 .type(ProviderConfigProperty.STRING_TYPE)
                 .add()
 
                 .property()
                 .name("attribute-regex")
-                .label("Attribute RegEx")
+                .label("Validation Regex")
-                .helpText("A RegEx Rule used to verify each attribute value. Only matching values are returned.")
+                .helpText("Only values matching this regex will be included in the result. Optional.")
                 .type(ProviderConfigProperty.STRING_TYPE)
                 .add()
 

attribute-endpoints-provider/src/main/java/de/ccc/hamburg/keycloak/attribute_endpoints/AttributeEndpointsResourceProvider.java

@@ -18,6 +18,7 @@ import java.util.Collection;
 import java.util.List;
 import java.util.Map;
 import java.util.regex.Pattern;
+import java.util.stream.Collectors;
 import java.util.stream.Stream;
 
 public class AttributeEndpointsResourceProvider implements RealmResourceProvider {
@@ -66,6 +67,35 @@ public class AttributeEndpointsResourceProvider implements RealmResourceProvider
         return Response.ok(attribute_list).build();
     }
 
+    /**
+     * Return a map of lists of attribute values. For each attribute in the named attribute
+     * group, add an entry in the resulting map with the attribute name as the key, and the
+     * values as a list as the values.
+     *
+     * @param attributeGroupName attribute group name
+     * @return a map of attribute names and their values
+     */
+    @GET
+    @Path("export/{attributeGroupName}/map")
+    @Produces(MediaType.APPLICATION_JSON)
+    public Response exportAttributeValuesMap(@PathParam("attributeGroupName") String attributeGroupName) {
+        AttributeExportContext ctx = new AttributeExportContext(attributeGroupName);
+
+        List<UserModel> userList = ctx.users.toList();
+
+        Map<String, List<String>> attributeMap = ctx.attributeNames.stream()
+                .collect(Collectors.toMap(
+                        attributeName -> attributeName,
+                        attributeName -> userList.stream()
+                                .flatMap(user -> user.getAttributeStream(attributeName))
+                                .filter(attribute -> !attribute.isEmpty())
+                                .filter(ctx.filter::matches)
+                                .toList()));
+
+        return Response.ok(attributeMap).build();
+    }
+
+
     /**
      * Resolves and validates the configuration and request state needed to export attribute
      * values for a given slug, exposing the results as member variables.
@@ -87,37 +117,38 @@ public class AttributeEndpointsResourceProvider implements RealmResourceProvider
                     .toList();
 
             if (componentList.isEmpty()) {
-                throw new NotFoundException("Endpoint not found");
+                throw new NotFoundException("Attribute Endpoint " + slug + " not found");
             }
 
             if (componentList.size() > 1) {
                 throw new NotFoundException(
-                    "Endpoint Configuration Error - Multiple configurations exist for this endpoint.");
+                        "Attribute Endpoint Configuration Error - Multiple configurations exist for " + slug);
             }
 
             ComponentModel component = componentList.get(0);
 
             RoleModel authRole = realm.getRole(component.getConfig().getFirst("auth-role"));
             if (authRole == null) {
-                throw new ServerErrorException("Endpoint Configuration Error - auth-role does not exist.", 500);
+                throw new ServerErrorException("Attribute Endpoint " + slug + " Configuration Error - auth-role does not exist.", 500);
             }
 
             RoleModel matchRole = realm.getRole(component.getConfig().getFirst("match-role"));
             if (matchRole == null) {
-                throw new ServerErrorException("Endpoint Configuration Error - match-role does not exist.", 500);
+                throw new ServerErrorException("Attribute Endpoint " + slug + " Configuration Error - match-role does not exist.", 500);
             }
 
             upconfig = session.getProvider(UserProfileProvider.class).getConfiguration();
             configAttributeGroup = component.getConfig().getFirst("attribute-group");
-            if (upconfig.getGroups().stream().noneMatch(g -> g.getName().equals(configAttributeGroup))) {
+            if (upconfig.getGroups().stream().noneMatch(g -> g.getName().equals(configAttributeGroup)) &&
-                throw new ServerErrorException("Endpoint Configuration Error - attribute-group does not exist.", 500);
+                    upconfig.getAttributes().stream().noneMatch(a -> a.getName().equals(configAttributeGroup))) {
+                throw new ServerErrorException("Attribute Endpoint " + slug + " Configuration Error - no attribute or attribute group named " + configAttributeGroup + " found", 500);
             }
 
             try {
                 filter = new RegExFilter(component.getConfig().getFirst("attribute-regex"));
             } catch (Exception e) {
                 throw new ServerErrorException(
-                        "Endpoint Configuration Error - attribute-regex is not a valid regex pattern.", 500);
+                        "Attribute Endpoint " + slug + " Configuration Error - attribute-regex is not a valid regex pattern.", 500);
             }
 
             authUser = AttributeEndpointsResourceProvider.getAuth(session).getUser();
@@ -129,7 +160,9 @@ public class AttributeEndpointsResourceProvider implements RealmResourceProvider
             // select all attributes that match configAttributeGroup, or that are in a group that matches configAttributeGroup
             attributeNames = upconfig.getAttributes()
                     .stream()
-                    .filter(a -> a.getGroup() != null && a.getGroup().equals(configAttributeGroup))
+                    .filter(a ->
+                            a.getName().equals(configAttributeGroup)
+                                    || (a.getGroup() != null && a.getGroup().equals(configAttributeGroup)))
                     .map(UPAttribute::getName)
                     .toList();
 

attribute-endpoints-provider/src/main/java/de/ccc/hamburg/keycloak/attribute_endpoints/AttributeEndpointsResourceProviderFactory.java

@@ -1,13 +1,12 @@
 package de.ccc.hamburg.keycloak.attribute_endpoints;
 
+import com.google.auto.service.AutoService;
 import org.keycloak.Config;
 import org.keycloak.models.KeycloakSession;
 import org.keycloak.models.KeycloakSessionFactory;
 import org.keycloak.services.resource.RealmResourceProvider;
 import org.keycloak.services.resource.RealmResourceProviderFactory;
 
-import com.google.auto.service.AutoService;
-
 @AutoService(RealmResourceProviderFactory.class)
 public class AttributeEndpointsResourceProviderFactory implements RealmResourceProviderFactory {
     static final String PROVIDER_ID = "attribute-endpoints-provider";

attribute-endpoints-provider/src/test/java/de/ccc/hamburg/keycloak/attribute_endpoints/AttributeEndpointsResourceProviderTest.java

@@ -112,7 +112,6 @@ public class AttributeEndpointsResourceProviderTest {
         }
     }
 
-    /* NOTYET
     @Test
     public void exportAttributeValuesMap_doorisSlug_returnsSshKeysPerAttribute() {
         try (Response response = callAuthenticatedAs("export-dooris-ssh-keys",
@@ -125,9 +124,7 @@ public class AttributeEndpointsResourceProviderTest {
                     response.getEntity());
         }
     }
-    */
 
-    /* NOTYET
     @Test
     public void exportAttributeValues_mailingListChaosSlug_returnsAddressesOfAllMatchingUsers() {
         try (Response response = callAuthenticatedAs("export-mailing-list-addresses",
@@ -138,9 +135,7 @@ public class AttributeEndpointsResourceProviderTest {
             assertEquals(List.of("hacker+chaos@example.net", "tester+chaos@example.com"), response.getEntity());
         }
     }
-    */
 
-    /* NOTYET
     @Test
     public void exportAttributeValues_mailingListInternSlug_returnsAddressOfSingleMatchingUser() {
         try (Response response = callAuthenticatedAs("export-mailing-list-addresses",
@@ -151,9 +146,7 @@ public class AttributeEndpointsResourceProviderTest {
             assertEquals(List.of("hacker+intern@example.net"), response.getEntity());
         }
     }
-    */
 
-    /* NOTYET
     @Test
     public void exportAttributeValuesMap_mailingListChaosSlug_returnsAddressesPerAttribute() {
         try (Response response = callAuthenticatedAs("export-mailing-list-addresses",
@@ -166,7 +159,6 @@ public class AttributeEndpointsResourceProviderTest {
                     response.getEntity());
         }
     }
-    */
 
     @Test
     public void exportAttributeValues_unknownSlug_throwsNotFound() {

testing/import/testing.json

@@ -1605,7 +1605,7 @@
       "subComponents" : { },
       "config" : {
         "match-role" : [ "mailing-list-intern-member" ],
-        "attribute-group" : [ "mailing-list-addresses" ],
+        "attribute-group" : [ "mailing-list-address-intern" ],
         "auth-role" : [ "export-mailing-list-addresses" ],
         "slug" : [ "mailing-list-addresses-intern" ]
       }
@@ -1626,7 +1626,7 @@
       "config" : {
         "match-role" : [ "mailing-list-chaos-member" ],
         "auth-role" : [ "export-mailing-list-addresses" ],
-        "attribute-group" : [ "mailing-list-addresses" ],
+        "attribute-group" : [ "mailing-list-address-chaos" ],
         "slug" : [ "mailing-list-addresses-chaos" ]
       }
     } ]