Neuro-symbolic Predicate Invention: Learning relational concepts from visual scenes

3.1.Architecture overview

The NeSy-π is a neuro-symbolic system to invent high-level relational concepts as predicates. The invented predicates are further be used by NeSy-ILP system for target clause searching. The workflow of solving visual ILP problem using the NeSy-π system is illustrated in Fig. 2. The process is divided into two major components: the training and the evaluation. Training utilizes the training examples D={D+,D−} to search for target clauses C, which satisfies two conditions: 1) ∀A∈D+,C∪B⊧A and 2) ∀A∈D−,C∪B⊭A, as the target clauses and selected background knowledge should entail only positive images and no negative images. During evaluation, test examples will be classified as positive or negative based on the target clauses C returned during the training.

Fig. 2.

Workflow of visual scene reasoning (train, left) during the training phase, NeSy-π learns a set of rules C to describe common patterns within the given RGB-D images. This process involves using four models: a pretrained perception module (e.g. Mask R-CNN [11]) for object detection; a grouping module (GM) that fits objects to lines base on their 3D positions; NeSy-ILP as a rule learner; NeSy-π as a concept inventor. GM groups objects together (e.g. a group of objects aligned on a line) to form complex spatial relations (e.g. a check mark). (Evaluate, right) in the evaluation phase, given a test RGB-D image, the system predicts whether the image follows the set of rules C (positive) or not (negative). The GM and perception module are used for image processing, while a differentiable forward reasoner is employed for reasoning purposes.

As a prepossessing step, we prepared a dataset for training a perception module, such as Mask R-CNN [11]. The dataset for perception module is similar as training examples in D, which includes scenes with same object types but positioned randomly. Each scene is annotated with object labels and bounding boxes.

Once the perception module is trained, the four modules, perception module, grouping module, NeSy-ILP, and NeSy-π perform an end-to-end training, which takes training examples d∈D as input and reasons target clauses C. Firstly, the perception module takes the RGB image of training example d to detect the labels and the bounding boxes of the objects, then the 3D positions of each object can then be calculated by utilizing depth map of each example d, the camera matrix K along with the labels and bounding boxes from the output of perception module. After acquiring the object positions, for providing simple and high-level descriptions of the scenes, the Grouping Module (GM) combines objects into groups based on their positions and encodes these scenes as group tensors, which capture the geometric information of the group, such as dimension scales and positions. These group tensors are then utilized as input for the NeSy-ILP system, which iteratively generates and evaluates clauses on the group tensors to identify the target clauses C. In each iteration of clause searching, NeSy-ILP outputs the searched clauses and their respective evaluation scores to NeSy-π. NeSy-π then invents new predicates by taking disjunctions of the promising clauses and returns them to NeSy-ILP. By incorporating these new invented predicates, NeSy-ILP can use them in the subsequent iterations to search for new clauses with greater precision in describing the rules. The final output of the training session is the set of target clauses produced by the NeSy-ILP system.

During evaluation, the system classifies the test scenes as either positive or negative. Positive scenes satisfy the target clauses while negative scenes do not. The perception and grouping module first interpret the data into group tensors, just as during training. Then, a differentiable forward reasoner is used to classify the group tensors as positive or negative based on the target clauses C.

3.2.Clause extension

We take the top-down approach for ILP, where new clauses are generated by specifying general rules [26]. We extend clauses step by step by evaluating on visual scenes to prune redundant candidates. NeSy-π begins with the most general clause for the search, e.g. the initial clause C0 is defined as follows:

where the atom in(O1,X) consists of the 2-arity predicate in/2 and variables O1,X, which represents the confidence of the object O1 in the image X detected by the perception module. The atom target(X) provides the probability that the image X is positive based on the atoms in the clause body. C0 is the most general clause that assesses the object’s existence in the image without imposing any property constraints, such as color or shape.

The clauses set is extended incrementally, starting with the extension of C0 in the first step. Then a set of clauses C={C1,…Cn} is generated where each new clause extends the previous clause by an atom from the language L .22 For instance, by adding atoms color(O1,red) and color(O1,blue) to the body of C0, the clause set C={C1,C2} is generated as follows,

The clauses in C are then evaluated, as introduced in Section 3.3 and only the top-scored clauses are kept for the next iteration.

3.3.Clause evaluation

We now describe the clause evaluation strategy in NeSy-π and three novel metrics to evaluate clauses efficiently for predicate invention. Given the generated clauses (described in Section 3.2), NeSy-π scores each of them using positive and negative examples D={D+,D−}. Note that each example (di+∈D+ or di−∈D−) is given as a visual scene. For clear presentation, we prepare the following notation:

We introduce metric P-Measure, which is used to evaluate the performance of the clause on positive examples. The target clause has to be valid in all positive examples, thus the more positive examples that a clause satisfies, the better the clause is. On the contrary, the less positive examples that a clause satisfies, the worse it is.

P-measure The P-Measure assesses the validation of clause on positive examples, with μCP representing the P-Measure of clause C. This value indicates the cumulative confidence of positive examples that satisfy clause C with its value in the range [0,1]. Since all positive examples should satisfy the target clauses, higher P-Measure of C indicates better performance.

Similarly, we introduce metric N-Measure, which is used to evaluate the performance of the clause on negative examples. The target clause has to be invalid in all negative examples, thus the less negative examples that a clause does not satisfy, the better the clause is. On the contrary, the more negative examples that a clause satisfy, the worse the clause is.

N-measure The N-Measure evaluates the validation of clauses on negative examples. Given a clause C, its N-Measure is denoted as μCN, measuring the inverse of the cumulative confidence of negative examples that do not satisfy C. The value of μCN is in the range [0,1]. As the target clauses are not intended to be satisfied by all of the negative examples, a higher N-Measure of C indicates better performance. The N-Measure of C is defined as follows:

If none of the negative examples satisfy C, the N-Measure of C is equal to 1. Assuming the examples are either positive or negative, if there is an example e that satisfies C, then d is a positive examples. The clause C is called a sufficient condition for the positive examples, which follows the implicational relationship

If all the positive examples are satisfied by C, and no negative examples satisfy C, then C meets the definition of the target clause. In this scenario, both the P-Measure and N-Measure of C are 1, and C is a sufficient and necessary condition of the positive examples. This follows the implicational relationship:

However, the majority of the clauses yield P and N measure scores between 0 and 1, i.e. they satisfy neither necessary nor sufficient condition. Since the sufficient and necessary condition is satisfied by μCP(D+)=1 and μCN(D−)=1, the higher the clause resulting in both P-Measure and N-Measure scores, the closer it is to the target clause. NeSy-π utilizes a best-first approach that take the both P-Measure and N-Measure into account, resulting in the following PN-Measure.

PN-measure The PN-Measure evaluates a clause meeting the definition of the target clause. Given a clause C, its PN-Measure is denoted as μCPN, measuring the product of its P-Measure and N-Measure, which is defined as follows:

3.4.Predicate invention

NeSy-π invents new predicates by using the extended clauses C and their evaluations, which were described in Section 3.2 and Sec, 3.3, respectively. New predicates are defined by taking disjunctions of the extended clauses. We assume that all clauses in C have the same predicate for their head atom.

For example, assume C={C1,C2,C3} (|C|=3) as follows:

A predicate inv_pred_1 can be invented by disjunctively combining the first two clauses. That is, let Cp={C1,C2} denote the explanation clause set of inv_pred_1, then inv_pred_1 is defined as follows

The invented predicate inv_pred_1 interprets the concept that there exists a pair of objects in the image with the same color (either blue or red). Each clause within Cp is referred to as a variation. By taking the disjunction of the clauses, the invented predicate can be assigned true if any one of its variations is true. Similarly, another predicate inv_pred_2 can be invented by taking the disjunction of the last two clauses, inv_pred_3 can be invented by taking the disjunction of all three clauses, and so on. In total, there are 2|C|−1−|C| predicates that can be invented from a clause set C that includes at least two clauses in its explanation clause set. Due to the exponential growth rate, the invention of predicates may be restricted by a threshold, denoted as a. This threshold is defined such that 2⩽a⩽|C|, where |Cp|⩽a. As a result, subsets that surpass the cardinality of a are not utilized for predicate invention. Applying the threshold a, enforces that the possible variations of concepts are within the limit of a.

The key of predicate invention in NeSy-π is to identify the explanation clause set, which is a subset of the clause set C searched by the NeSy-ILP system. Due to the absence of guidance or background knowledge, the model lacks the ability to choose the optimal explanation clause set. Nonetheless, it is feasible to invent the clauses first and then evaluate them on a dataset. First, NeSy-π generates all possible clause combinations from the clause set C, then evaluates each clause based on the dataset and finally selects the top-k highest-scoring new predicates.

Now we move on evaluating the newly invented predicates. Invented predicates can be evaluated using the P-Measure and N-Measure metrics as introduced in Section 3.3. Let p be an invented predicate, Cp be the corresponding explanation clause set, and V_i its i-th variation.

P-measure on invented predicate The P-Measure of invented predicate candidate p is utilized to evaluate the validation of an invented predicate on positive examples. The validity of predicate p increases as more positive examples are verified to hold true on it. The invented predicate is defined as the disjunction of the corresponding explanation clause set Cp. The maximum score of the clauses in Cp determines the evaluation score of p, which is calculated as follows

N-measure on invented predicate The N-Measure is utilized to access the validation of an invented predicate on negative examples. The fewer the number of the negative examples in which the invented predicate holds true, the stronger its performance. The invented predicate is defined by the disjunction of the associated explanation clause set Cp, and its evaluation score is determined by the highest-scoring clause. The score of predicate p on example d is calculated as maxVi∈CprVi(d). To indicate that the higher the value, the better, the result is computed as 1−maxVi∈CprVi(d). It is also equivalent to minVi∈Cp(1−rVi(d)). The N-Measure for an invented predicate is determined in the following

PN-measure on invented predicate The PN-Measure assesses the performance of the predicate on both positive and negative examples in a quantified, objective way. This measure is derived by calculating the product of the P-Measure and the N-Measure. The formula for the PN-Measure can be written as follows:

In NeSy-ILP systems, target clauses are extended by searching for atoms within the given language. This search relies on the assumption that all the necessary atoms are present in the given language. In contrast, NeSy-π does not require such an assumption. If necessary atoms are absent from the language, the system can invent new predicates and generate new atoms.

For instance, let us consider scenes with only two colors red and blue and two shapes sphere and cube. The pattern three_same represents the concept that three objects have same color and shape simultaneously. A possible solution for NeSy-ILP system necessitates the 3-arity predicate two_same/2, which is presented as the background knowledge. The target clause can be then be obtained by extending two atoms from the initial clause

The predicate two_same is explained by five clauses, as shown in Listing 1, which are presented as background knowledge. In NeSy-π, such kind of background knowledge is not provided, but learned by the system. An example of rules that NeSy-π might learn is illustrated in Listing 2. By updating the invented predicates for the language, NeSy-π can search for the target clause as follows:

Listing 1.

Background knowledge of concept two_same

Listing 2.

Predicates invented by NeSy-π that relate to concept two_same

The predicate target(X) corresponds to the concept three_same. It takes five clauses to explain, which are learned instead of given. The predicates inv_pred1/2 and inv_pred2/2 are invented predicates. The other three predicates shape/2,color/2, and in/2 are given beforehand. Predicate invention enhances the system’s ability to learn new concepts with less reliance on background knowledge.

3.5.Object grouping

In a scene with numerous objects, it is often unnecessary to consider every relationship. For instance, if there are three objects on a table, it is reasonable to query whether any two of them have the same color or same shape. However, if there are 20 objects on the table, it is not necessary to investigate whether any two of them share a property. It would be more expedient to initially consider conditions such as whether more than half or all of the objects share a property. In order to gain a comprehensive understanding of an image, it is advisable to begin by accessing it at a global level, rather than delving into its particulars right away.

For complex patterns that comprise a multitude of objects, NeSy-π utilizes an object Grouping Module (GM) that can recognise line patterns in a more efficient and organised manner. The grouping module employs the position characteristics of each object to group them accordingly. It is essential to view multiple objects as a single entity in complex scenes since the number of object relationships increases at a factorial rate, specifically O(n!), with n being the number of objects in a scene. Examining relationships between different groups instead of individual objects can significantly decrease the number of relations under consideration. However, not all patterns can be fully described by line groups; rather, isolated objects must be treated individually.

Five steps are required to convert an image example to group tensors G. The main procedures of the grouping module are illustrated in Fig. 3. Additionally, the algorithm provides certain thresholds. ϵ denotes the distance threshold between an object and a group, whereas NG is the minimum number of objects required in each group.

Fig. 3.

Grouping module. The grouping module takes an image as input and produces the group tensors G in 5 stages. (1) the perception module detects objects and encodes them as tensors E; (2) objects are chosen based on their properties to generate small groups. (3) extend each small group with as many objects as possible. (4) evaluate and select the highest-scoring groups. (5) encode groups into group tensors G.

Object perception In the first step, the grouping module detects the object labels OLabel∈ZN×1 and the bounding boxes OBox∈RN×4 in the RGB image IRGB∈RW×H×3 based on an object detector, such as Mask-RCNN [11], where N is the number of objects detected, W and H are the width and height of the image. Based on the camera matrix K, the 3D coordinates of all the pixels I3D∈RW×H×3 relative to the camera coordinate system can be calculated from the depth map IDepth∈RW×H×1. Since in RGB-D images, the pixels on IRGB and IDepth correspond to each other, each bounding box in OBox corresponds to the same region in I3D based on the pixel coordinate system. This allows the object surface positions to be recorded relative to the camera coordinate system. For each object there is a set of surface points. The object position can be approximated by the median number of these points. For all the objects, their positions are given by O3D∈RN×3. Given O3D and OLabel, objects can be encoded into tensors E∈RN×M, where M is the number of properties for each object. In NeSy-π, the properties taken into account are position, color, and class. The position is specified by the 3D coordinates of the object, while both colors and classes are one-hot coded.

Group generation NeSy-π proposes an approach to gather objects into line groups based on their 3D positions. The strategy is to first generate small groups and then expand them to the maximum size. Since two points define a line, a small group is generated by two objects. The object positions can also be used to determine the corresponding line function. For N objects, N2 small line groups are generated. As shown in Fig. 3, seven objects are placed on the table in the given example, thus 72=26 small groups are generated from this scene.

Group extension The groups are further extended based on their line functions. For each small group, the remaining N−2 objects not contained in the groups can be evaluated based on their distances to the line. By specifying a threshold parameter ϵ, the objects with have distances within ϵ can be extended to the groups. As shown in Fig. 3, Obj1 and Obj2 can generate a small group. Let l12 be the line defined by this group, the system will calculate the distance between the remaining objects Obj3,Obj4,Obj5,Obj6,Obj7 to l12, and then compare the distances with the threshold ϵ. By setting a suitable threshold, Obj3 is extended to the group. Similarly, the small group created by Obj3 and Obj4 can be extended with Obj5,Obj6 and Obj7.

Group evaluation Since the motivation of grouping module is to decrease the relations between objects, it is important to minimize the number of groups. After the group generation, N2 group are generated and then expanded to their maximum potential. The group evaluation removes duplicate groups. Additionally, a minimum group size of NG is established, which serves as a system parameter to remove groups with fewer objects than the threshold. Choosing a smaller value of NG enables the explanation of rules in great detail, but at the cost of increased time and space requirements. Conversely, choosing a larger value of NG leads to faster reasoning, but pattern details may be overlooked. For our experiment, we opted for the smallest possible value of NG that still maintained acceptable reasoning times in order to enhance its robustness.

Group encoding Finally, the groups are encoded as tensors G. The position of each group is determined by averaging the positions of objects within the group.

The group module is an optional module of NeSy-π. It is used for scenes with potential group structures, like lines in this given example. By generating rules at group level, the logical structures between objects can be expressed much more clearly and easily. The invented predicates are also invented at the group level.

Algorithm 1

Predicate invention on visual scenes using NeSy-ILP and NeSy-π

3.6.NeSy-π algorithm

Algorithm 1 provides the pseudo code of the process of predicate invention utilizing the NeSy-ILP and NeSy-π systems. The NeSy-π system focuses on extending the initial language L0 to the output language Lout through the invention of new predicates. Meanwhile, NeSy-ILP is tasked with searching for target clauses.

The algorithm includes a grouping module as discussed in Section 3.2–3.5. The initial language L0 includes all the background predicates alongside the initial clause C0, namely target(X):−in(O1,X). Additionally, the camera matrix K, image dataset IRGB and the corresponding depth map IDepth are also inputs. The algorithm’s output is the extended language Lout, which includes all invented and background predicates.

(Line 1–4) Produce object tensors E from the input dataset IRGB and IDepth. An object detector, like Mask R-CNN [11], is employed to identify object labels OLabel and bounding boxes OBox in the RGB images IRGB. The 3D coordinates of all pixels I3D can then be transformed, relative to the camera’s coordinate system, from the depth map IDepth and the camera matrix K. The 3D position of the objects O3D can be determined by taking the median value of the corresponding region in I3D with respect to corresponding bounding box OBox. (Line 5–7) Group the objects based on their positions O3D. (Line 9–12) The iteration variable N is responsible for specifying the number of groups covered by the searched clauses. The given parameter Nmax_groups controls the maximum number of groups covered by the clauses, which is set with respect to the complexity of the training patterns. For complex Kandinsky Patterns, this parameter is set to 2, while for Alphabet Patterns, it is set to 5 or 6, as detailed in Section 4.1. Additionally, two lists are initialized. Pinv_all serves as a repository for invented predicates, while Pbk is the set of all background predicates in the initial language L0. (Line 14–17) In each iteration, a background predicate P is selected from Pbk. A new language L is instantiated with no pre-existing predicate. P is the first predicate that is used to update this language. (Line 19) The for-loop spanning lines 19–38 comprises the clause extension (line 22–29) and predicate invention (line 33–37). Nstep_max determines the maximum length of the clauses that can be extended. (Line 22–31) Extend the clauses in the clause set C for Nstep_max iterations. During the first iteration, the clause in C serves as the initial clause. The clauses are subsequently extended with new atoms, followed by the evaluation of the whole dataset. The top-scoring clauses are retained while the rest is pruned. If a target clause has been identified, then the for loop is terminated. (Line 32–37) After the clause extension, invent the predicates. Each element in C is a subset of extended clauses C, as well as an explanation clause set. The new predicates Pinv are generated from C. The evaluation of the invented predicates is based on the clauses in the corresponding explanation clause set. The score of these clauses can be obtained directly from μPN. By limiting the number of invented predicates, only the top-k invented predicates are retained. The invented predicates are then added to the list Pinv_all and the language L. (Line 44) The algorithm returns a language Lout consisting of all the remaining invented predicates from Pinv_all.

4.1.Datasets

The datasets comprise synthetic images generated using the Unity game engine [35]. In every image, objects are positioned on a table and captured by a calibrated RGB-D camera. A scene in the dataset refers to an image captured by the camera. Every scene is made up of basic objects that represent logical concepts. The object categories comprise of three types: sphere, cube, and cylinder; the objects come in three colors: red, green, and blue. The size and surface texture of the objects remain consistent across all scenes. A pattern refers to a set of scenes consisting of both positive scenes and negative scenes. All of the positive scenes follow a common logical concept, while none of the negative scenes do. To examine the predicate invention performance of the NeSy-π, every pattern in the dataset is accompanied by at least one new concept not included in the initial language.

Fig. 4.

Simple Kandinsky Patterns. These patterns are designed with only a few objects. Each pattern refers to a logical concept, as indicated by the name, which is not explained by background knowledge. Close: two objects are close to each other. Diagonal: two objects are diagonally located. Same: three objects have the same shape and the same color. Two Pairs: two pairs of objects are in the scene. A pair is defined by two objects with the same color and shape.

Simple Kandinsky Patterns This dataset represents four logical patterns, as shown in Fig. 4. The patterns comprises a maximum of four objects. In each pattern, several objects are placed on the table. The details of each pattern is introduced as follows:

Fig. 5.

Complex Kandinsky Patterns. These patterns are designed with at least five objects in the scenes. These objects form 2D shapes by their position. Check Mark: several objects form the shape of check mark. Parallel: multiple objects are arranged in two parallel rows. Perpendicular: multiple objects are arranged in two perpendicular rows. Square: several objects are arranged in the shape of a square.

Complex Kandinsky Patterns This dataset uses objects to represent four logical patterns, as shown in Fig. 5. Compared to the simple Kandinsky patterns, the complex Kandinsky patterns consists of a considerately larger number of objects and relations among objects. The objects are ordered according to certain spatial rules, such as perpendicular, parallel and so on. The details of each pattern is introduced as follows:

Fig. 6.

Latin Alphabet Patterns. Latin letters (including upper and lower case) consisting of multiple objects. The shape of the individual letters consists mainly of lines. For more examples, see appendix B.

Latin Alphabet Patterns This dataset uses objects representing 26 Latin letters. Figure 6 shows some examples of the letter patterns. A complete version is shown in the Appendix B. These patterns are composed of line-based elements. In each pattern, a set of objects is placed on a table and forms the shape of a letter in the Latin alphabet. In order to introduce more variations for each pattern, letter cases (upper case and lower case) are considered, i.e. each pattern has two variations. The colors and shapes of the objects are changed randomly in each scene. The center position of all objects is also shifted slightly in each scene. The dataset of each pattern consists of the same number of positive and negative scenes. The positive scenes are created based on the two variations of that pattern, and the negative scenes are created based on the variations of all the other patterns.

4.2.Evaluation results

The experiments aim to answer the following questions:

A1 (NeSy-π effectiveness): The background predicates utilized for solving the patterns in this work are listed in Table 1. All predicates invented by NeSy-π are based on these predicates. Moreover, no other predicates or background clauses are provided for the reasoning process. In αILP [30], certain concepts such as same_color,same_shape are explained using a set of background clauses, but no such clauses are provided for reasoning in NeSy-π. However, certain background predicates necessitate parameters determination for precision purposes. The parameter Nρ decides the precision of the distance predicate (rho/3). For example, Nρ=20 means that the distance between two objects can be characterized from rho0 ([0.00W,0.05W]) to rho19 ([0.95W,1.00W]) with W as the table width. Similarly, the precision of the direction predicate (phi/3) is determined by the parameter Nϕ, while the precision of the slope predicate (slope/2) is determined by the parameter Nslope. These parameters determine the granularity of the concepts that can be handled in the system.

Figure 7 shows an example of the learning result for the simple Kandinsky pattern close. The target clause is searched by αILP, and the used predicate inv_pred253 is invented by NeSy-π, which can be interpreted by the distance between object O1 and O2 is in the range rho4 to rho8. The invented predicates that are not used in the target clauses are not listed in the figure. Appendix B shows the corresponding target clauses and the invented clauses of the rest patterns.

Table 2 shows the evaluation result of all simple Kandinsky patterns based on the proposed metrics in Section 3.3. NeSy-π has successfully invented new predicates and found the promising target clauses for all simple patterns, resulting in PN-Measure values over 100% higher than the baseline model αILP, with the exception of the diagonal pattern at only 14.94%. This is due to the limited variations in the diagonal pattern compared to others. Only two variations exist in the positive scenes (since a square has two diagonals), and two variations exist in the negative scenes (forming a row either at the top or bottom of the table). Other patterns, such as close, have randomly changed object distances within a specified range. Consequently, there are many more variations in these patterns. PN-Measure is a balanced measurement that considers both positive and negative examples simultaneously. Therefore, the outperforming of PN-Measure does not guarantee the outperforming of both P-Measure and N-Measure.

Fig. 7.

Pattern Close. The logic of the pattern is that two objects are close to each other. Left: two positive (green border) and two negative (red border) examples of the pattern Close. Right: learned target clauses and the corresponding invented predicates of the pattern Close. The distance precision parameter Nρ is given as 20. The predicate inv_pred253 evaluates if the distance of O1 and O2 in range rho4∼rho8 (15%∼40% of table width).

Fig. 8.

Pattern Check Mark. The logic of the pattern is that several objects form a shape of check mark. The number of objects on the right side of check mark changes randomly from 3 to 5 in each scene. Left: two positive (green border) and two negative (red border) examples of pattern Check Mark. Right: learned target clauses and the corresponding invented predicates of the pattern Check Mark. The predicate inv_pred0 evaluates if the direction of O1 and O2 in range phi3/phi4/phi10/phi12.

A2 (Complex Patterns): For the complex Kandinsky patterns, NeSy-π uses the Grouping Module to gather the objects into groups. The predicates are invented on the group level. In the experiment, the maximum group number of the grouping module, i.e. the parameter N in Algorithm 1, is given as 5. The dataset Complex Kandinsky Patterns uses objects representing basic line relations and other basic geometry shapes. The given background knowledge is the given predicates shown in Table 1, and the line grouping module explained in Section 3.5.

Figure 8 lists the learned target clauses and the corresponding invented predicates of the pattern check mark. The target clause is described by 2 invented predicates. The used invented predicate inv_pred0 is explained as follows: the group O2 is either in the direction phi3, phi4, phi10, or phi12 of O1, which corresponds to the direction ranges [36∘,54∘], [54∘,72∘], [162∘,180∘], and [198∘,216∘] respectively. The predicate inv_pred211 is explained as follows: the distance between two groups is in the range rho4 to rho7, which corresponds to the distance ranges [0.15W,0.35W] with W as the table width. Appendix B shows the learned clauses of the remaining three complex Kandinsky patterns. Table 3 shows the scores of each pattern based on proposed metrics. NeSy-π improves the scores in both P-Measure and N-Measure with using 1–2 invented predicates in the target clauses.

Furthermore, we have also analyzed the more intricate dataset Alphabet Patterns, which consists of 5 to 20 objects. The learning result of certain letters on NeSy-π with αILP systems are shown in appendix B. Table 4 illustrates the best learning result of each patterns on two systems. NeSy-π shows an average PN-Measure score of 0.911, exhibiting a 6.55% improvement compared to αILP. The average P-Measure is 0.942 and N-Measure is 0.987. Certain letters achieves even 20% to 30% improvement on PN-Measure, such as letter I, L, M and R. Some letters achieves only minor or even negative improvement, that is because the baseline model αILP already achieves high scores (mostly higher than 0.9), thus the improvement spaces left to the NeSy-π are limited, such as letter D, H, K. In addition, NeSy-π does not guarantee a higher PN-Measure comparing to αILP alone, such as letter C. That is because the clauses are extended in a best-first approach in each step. The predicates in two systems are different, thus the best atoms for extension in two systems can be different. It is possible that a clause with invented predicates achieves high score in the previous steps, but increases only minor score in the following steps. In this case, the target clauses still contain the invented predicate, but its measurement can be smaller than the clause without invented predicate. In total, there is 1 pattern that does not invent any predicate, 11 patterns invent one predicate, 9 patterns invent two predicates and 5 patterns invent three predicates. On average, 1.65 new predicates are invented to describe each pattern.

A3 (NeSy-π efficiency): NeSy-π can improve the time efficiency for αILP system. The invented predicates improve the expressiveness of the language so that the rules can be expressed by shorter clauses. Consequently, the algorithm needs fewer iterations to search the target clauses, thereby improving the time efficiency.

In Fig. 9, the graph displays the runtime for each alphabet pattern. The height of each bar represents the corresponding letter’s run-time, with the color indicating the number of groups N covered by the respective target clause. The fewer groups the target clause covers, the fewer iterations the algorithm requires. Figure 9 (c) and (d) show the ranking of the training time in descending order. On average, the NeSy-π+αILP method requires 64 minutes and 47 seconds to solve one letter, while the αILP method takes 126 minutes and 28 seconds to solve one letter. Therefore, NeSy-π is roughly twice as fast. On the other hand, it is apparent that fewer iteration N adheres to the less training time. The letter B has the highest iteration and also the longest training time. On the other hand, letters such as U, X have the fewest iterations and, as a result, have the fastest training time. On average, αILP with NeSy-π module requires 0.46 fewer iterations than αILP alone.

Fig. 9.

Time consumption on Alphabet Patterns. Time consumption of the individual letters in descending order. The red dashed lines show the average time for each plot. The colors of the bars differ according to the number of groups covered by the target clauses. (A). Grouping time of αILP. (B). Grouping time of αILP with NeSy-π. (C). Training time of αILP. (D). Training time of αILP with NeSy-π.

Fig. 10.

Examples of robustness test dataset. The top row displays a selection of the original letter patterns. The bottom row shows negative letter patterns with one object randomly removed. To solve this dataset correctly, the model needs to be robust to the falsifying negative scenes not appeared in the training phase.

Fig. 11.

NeSy-π is robust to unseen falsifying examples. The evaluation of Alphabet pattern’s robustness is presented. Left: the NeSy-π performance in distinguishing negative examples is evaluated on two test sets. The purple line is based on a test set with dissimilar patterns as the negative examples, where different letters are used as negative examples. The orange line use a test set with similar patterns as negative examples, where one object is randomly removed from the same letters used as negative examples. Right: the performance of αILP and NeSy-π on the test set, where negative examples using similar patterns as positive patterns. The N-measure of NeSy-π achieves a higher average score compared to αILP.

A4 (NeSy-π robustness): In this experiment, we show that NeSy-π is robust to falsifying patterns that have not seen in the training phase, i.e. NeSy-π exhibits robustness in distinguishing similar positive and negative examples in the dataset, achieving high accuracy despite their similarity.

Dataset. In order to evaluate the robustness of the learned clauses, another test dataset for the alphabet patterns is generated with similar positive and negative scenes. The positive scenes are still generated from the original letter patterns. However, the negative scenes are generated by intervening the positive scenes, i.e. randomly removing one object from the positive patterns, as shown in Fig. 10. Such negative patterns are not seen by the model during training. To solve this task correctly, models need to be keen on small violations of the learned logical concepts.

Result. Fig. 11 shows the evaluation result for the robustness test dataset. In comparison to the efficiency result in A3, when negative examples with similar patterns as positive examples are used, NeSy-π performance on N-Measure decreases from 0.98 to 0.77. Additionally, none of the test image patterns were seen by the model during the training period, and they are also very similar to the positive patterns (with only one object removed). In addition, when the robustness test dataset is applied to the clauses acquired through αILP, the N-Measure average is approximately 0.70, as demonstrated in the right plot of Fig. 11. In this case, the NeSy-π attains a higher N-Measure.